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ABSTRACT . , 

This report describes recent- efforts to modifj^ test, 
and evaluate an experimental online database consisting of ^5 
Vendor- supplied, controlled subject vocabularies or thesauri. The 
experimental database, called the Vocabulary Switching System (VSS) , 
is designed' ta enhance search strategies and ultimately retrieval 
performance for users of online bibliographic datalmses by 
integrating vocabularies into common VSS files, thereby allowing the* 
user access t.o about 315,000 possible search terom. VSS assists .users 
with free-text or controlled vocabulary searches and single or 
multiple database searches. Work described includes (1) evaluation 
of numerous switching Strategies against the 15 residant vocabularies 
fOt VSjS; (2) controlled ^xperia^nts involving end-users and 
intermediaries; (3) system redesign and current status;* (4) system 
testing and evaluation; (5) results of a 1983^survey of online users 
and comparison with a similar 1979 survey; and (S) conceptual designs 
for a future switching system. Numerous tables, figures, and 
appendices are provided. (TBC). « 
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This report describes work perforaed under NSF. 6rant Itos. IST- 
79111^ and. IST-81U497, and covers' the periods October 1, J979 to 
Septeater 30, 198l' and January 15, 1982 to October 31, 1983, respectively. No 
attenpt Is nade to report, chronologically, the activity under these two 
efforts. Ilowever, the work performed can be generally described as follows: 

197^^1981 . Major software upgrade; expansion of experimental system from 6 
to 15 controlled vocabularies; evaluation of var1(His experimental 
syltchlr^ stacks; conc^tuallzatlon of expression - level swltch- 

1982-1983 Major feature 1ii<)lewentat1on (related terms); new user Interface 
(menu access); .field evaluation of VSS; update of 1979 on-line 
survey* . 
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EXECUnVE SUIMARY 

TTie Vocabulary Swttchlng Systas (VSS) Is an experimental system 
designed to enhance search strategies and ultimately retrieval perflhiianqe for. 
those^who use online bibliographic data bases. VSS contains 15 Indexing and 
retrieval vocabularles.f^m 12 /different suppliers. By fu\ly Integrating these 
vocabulflf'les Into comaon VSS files' the VSS user has access to about 315^ 
thousand passible search teris. VSS assists, users . wfth free-text or 
control led-vocabular^ searches and single or*lwlt1ple data base 'seanches. 

This rewJfT^^scrlbes research efforts undectaken to; (1) modify and 
expand VSS from an earlier more primltatlve version, and (2) evaluate VSS In 
end-user and IntenJfelary coiMunlties In real-Hfe situattons- In addition, a 
1979 surv^ of online users i«i|S updated td determine if- any shifts or -trends 
have occurred In user patterns or ^preferences over tltfe. 

During, th'ls research ,the^ systaa wa.s expanjded from/ 6 to 15 
vocabularies with each vocabulary being assigned to one or more of four 
modules. The VSS modules and vocabularies are: * ^ ^ 

Physical Sclaice KbduW; ChemtcCT • Afistracts* Concept Edit 
File; department of &ergy Thesaurus; Subject »Hfta.d1ngs for 
Eng1ni»eV1ng; Inspec. Theisaurus; Iron Center Thesaurus 
* . (tr'ntngyali, and NASA Thesaurus. - ' ; ^ 

Life Scleace Wp<faile: BIOSIS Master Index Authority File (2 
files* ci^at^r Hedfcal 'Subject leadings (3 files 
created): Chemical Abstracts Concept Edit File. • .» 

Social SclCTce Hodule; . ERK '\thesauruV. . Psycologlcal 
Abstracts Thesaurus. * 

tolnesx Waifale;' ABI Inform Thesaurus; Management- Contents 
Thesaurus. - 

Also during this period, VSS was modified to handle relat^cf- terms 
and both sanitized and unsanlt ized versions of all lead terms- Finally, a new, 
menu-driven interface was designed for use In the-evaluation phlse. 

End User evaluations of VSS were conducted a^ three separate remote 
sites using two -jdifferent evaluation methodologies. VSS perfoni«d quite well 
at one sit^ using an iterative methodology (one ^search -Intermediary for each 
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user query) but 1t was only-marginany better than the non-VSS searches at two 
other , sites using a parallel laethodo logy (two search Intermediaries for each 
user querjf). T>ie parallel niethodology seemed to t)e tf)e' least i|el1able of the 
two methodologies because seajrch 4ntermed1ar1es tended to Interpret the 6nd 
user query differently. 

Some i^ts professional search Intermediaries also evaluated VSS 
Independently of en^ users,* The participants thought VSS was easy to learn, 
use and understand and, quite naturally, rat'ed the system's performance In 
direct proportion to the amount of usable output It generated for theli^ query. 
Ratings of 3.5 to 4.1 on a S-^oInt scale were achieved- when the amount of 
usat)le output reached 6 or more terms. - 

« About 4 1/2 t^mes as many participants thcHjgfit MCSS would make tjieir 
JoB^ easier as thought VSS would make their Jobs harder, ^out 3/4 of all 

S>art1c1pants thought highly of the concept ^ of subject switching as a search 
tool, and* about l/t of all participants thought VSS was very valuable, 

^Valuable, qr "Interesting but needs more work". It was concluded that subject 
switching has a potentially wide Sppeal ' to Information professionals but VSS 
Itself ^eeds more work to address their needs and concerns. 

The online user's survey ccftiflrmed reported growth figures fbr the 
online .-Industfy, that'' Is, gi*owth at a 30 to 40 percent rate corapounded 

* 

annually. Also, fend users are paying for a greater percentage of their own 
searches in 1983 ciapafed to 1979. The ijisage pattern of seven major retrieval 
sei?v1ces showed little chan^ over the four year period, J979 to 1983 and ^he 
survey showed Hhat multlpTe data -base searching Is Increasing with time In the 
direction of 4 or more d^ta bases per search request. Results suggested that a 
user's need. for VSS 'Increased as their need for multiple data-base searches 
incretased. 

It se&ns certain that user accep-tance of VSS will be quite high If 
..the system contains (1) more vocabularies (2) more of the syndetic relation- 
ships, available In the host vocabularles^i (3) a means of attaining an4 
maintaining the most current ver'slons of host vocabularies, and (4) a more 
direct approach tg siearfchlirg VSSU These fmproyements are all " i«ll- w.lthin the 
state-of-the-art of data base' and thesaurus technology. . 



■V ■ 



TABLE OF CCWTEMTS 

• V Page 

1.0 INTROWJCTION ...i.../. * ^^ 

2.0 BAOCSROUND v •••• 2 



3.0 PURPOSE • *• 

4.0 OTJECTIVE^ AND SCt^E" ...... X- 

5.0 OVERVIEW VSS (PRESEriT VERSION) 9 

5.1 VSS Vocabularies » 

5.2 Inverted F13es * }3 

5.3 Conceit File J' 

• 5.4 System Coanands > ^ 

5.5 Switching Options , ^ <•* 

5.6 Using VSS : ; ^7 

6.0 VSS RESEARCH ACTIVITIES '. - ^ 

6.1 Software Developaent '. ; 23 

6.2 ^System Evaluation 23 

7.0 EVALUATKW 'OF a<ITCHIf» STRATEfilES ....... .V.. 24 

7.1 .Methodology * 

1 7.2 Results tv.---'- •••••• 

8.0 END USER EXPERIMENTS 4l 

8.1 Methodology %....•..;!.,. 41 

8.2 Results ;^ *. >....:....*..^;. 44 

g.O EVALUATIWI op" VSS BY INTERICDIARIES j.. 57i 

9.1 Methodology 57 

9.2 Results * 58 

9.2.1 Characterization of the Study Participants 58 



9.1.2 Correlation Between Participant Deraographlcs and - 

\ Overall VSS Evaluation i .' 5SL 

9.2.3 Correlations Between VSS Performance and User ^ 
Proficiency or User Individuality . $7 

9.2.4 Sifflmary of Open-ended ^Juestloas - 70 




10.0 ON-LINE USERS SURVEY OTATE ...... ^ 

10.1 Methodology 72 

10.2 ResuTts ^ 73 

11.0 CONCEPTS FOR EXPRESS IWIrLEYEL SWITCHING ..^f*. 91 

1:2.0 SlIWARY AND COICLUSIONS . . ^ 93 

12.1. Controlled Vocabularies , 93 

12.2 Multiple Data-Base Stfm:h1ng 93 

' 42.3 The VSS System ....V. ^ 94 



o > f ^ - 

ERIC / • . ; . - • . 7 



TAK£ OF cgrrarrs 

(Continued) 



12.4 Performance of^ VSS Modules 

12. 5 End User Ixpef'loents 

12.6 Evaluation df VS$. by Intensedlarles 

12.7 On-Llne Users Survey — 



13.0 . ^lECdNiCNOATIOIIS 
14.0 REFERENCES ... 



95 
96 

•98 
100 

loar 

.104 



LIST OF imJES 

TABLE 1 VSS VOCABULARIES , 11 

TABLE 2 VSS FILE SIZE BY VOCABULARY . '. 16 

JABLE 3 AVAILABLE SYSTDl COlMAi«)S *. 18, 

TABLE 4 AVAILABLE »ITCHII« »»TI(mS f. . . .4. • ... 20 

Table 5 perforjwice of ,vss wmES. * - 28 

TABLED performance OF BUSINESS l«0ULE * 31 

TABLE 7 PERFfflWANCE OF SOCIAL SCIENCE Wmi J. 31 

TABLE 8 PttFORMANCE OF SMALL IIFE SCIENCE MODULE 32 

TABLE 9 PERFORMANCE OF MEDIUM LIFE SCIENCE MODULE 32 

TABLE 10 PERF0WIWK:E LARGE LIFE SCIEJKE MMHJLE ./ 33 

TABLE 11 PERF0RMA»«:E PHYSICAL SCIEICE MKHiLE ......^ 33^ 

TABLE 12 DETAILED' PERFORMANCE dF IWJIVIWIAL SWITCTIf^ STRATEGIES - 35 

TABLE 13* EXPERIMENTAL DESIGN FOl Tl€ PARAULa TEST INVOLVING VSS 42 

TABLE 14 SUMMARY OF El© USER TEST? AT VARimiS TEST SITES 48 

TABLE 15 DETAILED SEARCH RESULTS FOR TE5T SITE 1 

QUESTKW (A) : 49 

TABLE 16 DETAILED SEi«CH RESULTS m TEST SITE 1 - . . . 

QUESTION (B) ^ 50 • 

TABLE 17 DETAILED SEARCH RESJLTS FOR TEST SITE 2 - ' - * 

QUESTI(» (A)^ ^ J...., , 51 

TABLE 18 DETAILED SEARCH RESULTS FOR TEST SITE 2 - - 

QUESTION (B) , v.. 52 

TABLE 19 DETAILED SEARCH RESULTS FOR TEST SITE 3 — • 

QUESTION (8301 J ^ 53 

TABLE 20 OETAILEI^'SEARCH RESULTS FOR TEST SITE 3 - 

QUESTItW (8302) .- ..^ * 54 

TABLE 21 DETAILED SEARCH- RESULTS FOR TEST SITE 3 — 

QUESTKMi (8303) • 55 

TABLE 22 ♦ DETAILED SEARCH RESULTS FOR TEST SitE 3 - 

QUESTW (8304) 4........ 56* 

TABLE 23. BRE/^COOMN OF.STDDY PARTICIPANTS BY EMPLOYMENT SECTOR ' 59 

TABLE 24 DISTRIBUTION Of RARTICIPANTS' PROFICIEICY T(« THE 

VSS DATA BASES 60 

TABLE 25 USER EXPERIEIttE VERSUS IMPACT OF VSS OH A 

- SEARC»«R»S JOB / i.. 62 



ERIC 



8 



lUmJE. OF CatTEWTS 
(Continued) 



TABLE 26 

TABLE 27 

TABLE 28 
JABLE 29 
TABLE 30 
TABLE 31 
TABLE n 

TABLE 33 
TABLE 34 
TABLE 35 
TABCE 36 
TABLE 37 

TABLE 38 
TABLE 39 
TABUE 40 
IPLE 41 



FIGURE 
FIGURE 
FIGURE 
FIGimE 
FIGl^E 
FIGURE 
FIGURE 
FIGimE 8 
FIGURE 9 
FIGURE 10 



1 
2 
3 
4 
5 
6 
7 



tlST^ TABLES 
(Continued} 

If«»ACT-(S^ YSS.ON TO PERFOimANCE BY 

EMPLOYJCNT SECTOR 

DISTRIBUTION OF SEARCHES BY USABLE VSS OUTPUT BY 

SWITCHirte STRATEGY . i 

VSS Wings by iw®er of ojtput terms by wms 

ESTIMATED AVERAGE RATII^ FOR VSS SEARCHES BY imA 

INDIVIDUAL SEAROI /WTTIVITY .>« 

CORPORATE SEARCH ACTIVITY % 

ON-LINE SEARCH ACTIVITY AND EXPENDITURES BY 

USER SECTOR (1979) 

MHO PAYS FOR ON-LINE SEARCHES , v 

RESTRICTKWS PLACED (Ml ON-LINE SEARCHES 

ONLINE RETRIEVAL SYSTEM USAGE ... 

REASONS GIV^N FOR SEARCHING A SII^E DATA BASE , 

SUBJECT AREAS MOST USEFUL F(» MULTIPLE DATA ^ 

BASE SEARtXir^ - 

USER PREFERENCES FOR TYPE SJBJECT SEARCHING DONE .4...... 

ANTICIPATED l^E. VSS 

MI6RATI(» TO WLIIPLE I^TA^E SEARCTING VIA VSS 

U^tR'S WILLINGNESS TO RAY F^ S^JECT 'S<IT(31If« ^^.v*, ........ 

J I LIST OF noigs 

MENU-DRIVEN MODE OF VSS 

•VSS TERMINAL SESSION IN MEmj-MODE 

USER'S RATINGS OF VSS 

VSS PERF(»»IAHCE AS fCASl«ED BY USEFOL OUTPUT ; 

INDIVIDUAL ON-LINE SEARCH ACTIVITY ; 

CORPORATE ON-LINE SEARCH ACTIVITY 

l»tlLTIPLE DATA BASE SEAROtlNG PATTERNS 

USAGE OF v(k:abularies ' , 

. WISER W^, responses /. 

rawBER of responses 



All'ENOIX A 
APPENDIX B 
APPENDIX C 
APPENDIX D 
APPENDIX E 

APPENDIX F 
APPENDIX G 
APPENDIX K 



LIST OF AITOPICES 

VOCABULARY PROCESSUS AND VSS imST FORMAT : 

VSS FILE BUILDING ROUTINES AND RECORD STRICTURES 

VSS OPERATING INSTRIK:TI0NS i. 

TERM SETS fOR SHITCHII^ STRATEGY ANALYSES 

EVALUATKW PROCEDimES AND F(«MS JISED AT TEST SITES ^ 

2 AND 3 y..... ..: 

VSS EVALUATKW FORM *. 

SUmARY ^OPEN-ENDED QUESTIONS 

SAI^LE QUESTItWNAIRE FOR (H«.INE USERS SURVEY 



1.0 mntoPUCTiow 

• » 

This report describes recent effprts to modify, test and evaluate an 
experimental online data base cwisl sting of 15 vendor-supplied, controlled 
subject vocabularies or thesauri. The experimental data base^ is called the 
Vocabulary Switching System (VSS). 

The work described herein Includes: * . 



• Evaluation of numerous switching strategies against the 
15 resident vocabularies In VSS 

• Controlled experiments Involving' end-users and Inter- 

• medUrles 

t System redesign and current status 

• System testing and evaluation 

• Results of a 1983 survey of online users and coi^arlson 

• with 1979 data " ^ ' " . . 

• Conceptual designs for a future switching sj^stem* 
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2.a B AdCfiROUIP 



Mvi*S stored base of ccwputer searchable bibliographic records has 
reached enonaous size. According to Cuadra (1) there are over 2,000 online 
daU' bases worldwide, about half of i*hich^re bibliographic or textuaU ^The 
bibliographic data bases ' alone contain hundreds^ of aillions of citations. 
DIALOG, one of the largest online services for bibliographic data bases, 
contains wore .than 170 data' bases and over 75 ail Hon records as of January, 
1983. Their largest database contains over 7 a^mion citations; the smallest 
aljout 4 thousand (2). ♦ * ^ . 

Other large online services include BRS (over 70^ data bases)^ 
Pergaaon Infoline (over ^ data bases and.wer 22 Billion records); SDC's 
ORBIT (over 70 data bases and ovw 55 aillioih citations); and ESA/IRS (over JO 
data bases anjl.. over 25 aillion ci tat tons). All toUlled, over 250, online 
serv.ices are available norldwide. • 

- With 'this scale as a reference, several (^servations can be aade: 



(1) Relevant infomation probably exists in soae online 
data base soaenhere on virtually any topic known to 

* aan " , / 

CZ) Locating the relevant inforaatlon oq -any .particular 
' topic 8«y >vary froa extreaely easy to virtually 
iffipossible and could be very costly depending on a 
nuffil^r of factor*: the user, the tc^ic, the 
intermediary (If used), the cmllne tervlce, the data 
base, the index, the nature of the data base records, • 
' the language of the ' source dociiraents,. how thjiii 
retrieval records are displayed (online or offline),- , 

• ' data t)ase and tele^offlMunicatldn" charges,' t1^ of day 
spent ;ieaiS6|i1ng, and moden baud rate, just to mention 
a few» 



(3) The' replac^ent of fisting Information retrieval 
technology (i.e., inverted files, free-text; and con- 
trolled-descrlptor indexijig. Boolean query formula- 
tion) with newer conceptst and technology is likely. to 
be evolutionary for naiiy yeafs ahead simply ^ause 
of the scale amf Investment In ^tbese ^f+stlng 
' systems. ' . ' . • - 



J 
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Bradford '5 Law of Scatter* Is easily understood and accepted by 
virtually everybody, in many cases Intuitively. If a conprehenslve. online 
search can be defined as locating a significant percentage of the relevant 
inforinatlon on a particular tc^lc, then, defending on how one defines^ 
significant, such a search usually translates Into a search of more than one 
•data base, in addition. If the seam:h Is a subject search, then a imiltl-fleld 
. search Is usually specified, Infcluding: (1) titles, (b) abstracts, (c) 
descriptors, (d) Identifiers, and'(e) subject' classifications. If they exist. 
These ^fields ire usually searched with a cjsifclnatlon of. free-text and" 
controlled descriptor terms although some searches use one or the other 

* * - * » 

approach exclusively. 

When more thin one data base Is Involved In a cwiprehenslve online 
subject search, dlffef-ln^ Indexing policies and -approaches Bwst be recognized 
If the search Is to be optimize^ for each data base. This Is lepeclally true 
if the '^search strategy Involves controlled descriptors. Several aids and 
^ approachesar^ available, either 'coranerclally or experlmentalj^to laprove 
aibject retrieval from online bibliographic data bases. ^ 

The most common search aid Is the master Index data base, a. file 
containing coiAlned Inverted Indexes of several data bases. The primary 
function of a master index data base is to guiile the user to appropriate 
specific data 'base(s) based on pasting frequencies for a stated topic. 
Several major retrieval services, have master indexes: 



lb' 



Retrieval Ulster 

Smice Index 

Dialog Oialindex , 

-BRS Cross 

SDC Data Base Index 

ESA/IRS (^jestindex 



♦Relevant information wilY be scattered oyer many sources, although a high 
' percentage will be .concentrated In just a few sources. 
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ThBS. while thd waster Hndex directs V user to certain ilata ba,ses, 
such an" 1n4.ex does^ little * to help the us^r state the query in the most 
appropriate system or Itwiixing language for* optlisaV retrieval under that 

topic, • ' . • ' • . ; 

A second aid Is the online thesaurus. Soic retrieval services permit 
a browse of technical thesaurus but K>nly for the daU base belrig sfearched. 
If multiple data bases * are to be searcM, each thesaurus must be searched 
-separately. If one «1sts, Hoiiever, BRS now dffiers an Integrated, controUed 
vocabulary data base ^pontain1ng subject headings fn» five social science 
vocabularies. This service, cafled TEWIS by WIS, 1^ similar to VSS. 

A third approach for 1n>,roved retrieval Is to raijk - order the 
retrieved citations according to some* Internally stored algorithm (e.g., 
co«b1nator1al\term weighting or pirctoablllty estimation). The user Is given a 
list pf &1tatlons^n ranked order where the highest weighted or most probably 
useful citations are presented first. The work of Ooszkocs and Cooper are 
exanples. of term weighting and probab111s.t1c approaches to Information 

retrieval, respectfully. 

Ooszkocs 6) has built a front «id to «ia.INE, TOXLJHE and other ft« 
data bases. This front end, called* Current Information Traiwfer tn English 
(C^TE), permits queries In natural Tanguajge, rartced output based ort 
octwtolnatorlal term weights, sf^i relevance feedback liwludl^ automatic query 
modification. . Term In the user's query are welghteii automatically by the 
system based on Inverse jpollectlon frequencies of such terms.^ ftitput Is 
ranked according to the* SMm of the weights of the query, terms contained In a 

given 'document. ' . 

- \ On the other hand. Cooper (4) nas pursued * probability theory In 
information rettjfval and presently bases his probability computation on the 
entropy of fllstr^blitli^ir" This approach Is called, the maxiwjm entropy principle 
%and claims that It overcomes the difficulties of other probability approaches 
because It avoids certalp Sl^llfylng assumptions concerning statistical 
Independence. Itawevec. he admits that Boolean retrieval Is so firmly 
entrenched In the relrleval world that even -front-ending" a maxirajw entropy 
system onto an^ existing Boolean system would take "some time" to undo the 
entrenchment. Thus, non-Boolean systems, although theoretically sound, are 
largely experimental and their acceptance seems to be some time off. 



In yet another approach, ^ already retrieved dpcufnent sets can be^ 
further analyzed for word frequency patterns on any fle^d stpeclfled by the. ^ 
user. The result Is a, list of words (occurring in the field specified) In 
d«;reas1ng'ord€r according to their frequency. Such' Lists provide additional 
clues to the user, for fine tuiMng 4 search; The European Retrieval Service, 
ESR/IRS, offenrs such^a search feature, called ZOOM. 

The ZOOM feature Is useful *fpr Identifying' terras ti*i1ch co-occur with 
those used In t^e original search strategy. It frees A user froa tiie task of 
actually reading abstracts, titles, descrlptbrs and otheir fields for 
additional clues with ' which to fine time a search. However, ZOOM cannot 
Identify co-occur1ng' words as synonjsBBs or*" g«ier1c terns and the results *a)-e 
d^endent on the quality of certain critical • fields (abstracts, titles, 
descriptors) and the miaber of citations being analyzed^ 

Other transparent Interfaces and linked systems provide partial 
solutions to the problem of accessing dlsslnllar files for similar^ 
infomatlon. The OWIT Systaa by Marcus (5) provides only truncated keyword 
aod full-phrase searching. The NIH-EPA tirtced chemical data base system (6) Is 
keyed exclusively to substances and chemical registry numbers (CRN). That Is,, 
once the CRN 1$ known, several' different files can be accessed ilnce they all 
use the CRN as access key. The Chemical Substances Information Jtetwork. like 
the MIH-EPA system Is keyA to chemical sObstance ,11r*ages (7). Again, this 
system operates ^n a limited domain. * ^ 

In the final analysis, linkage to dissimilar and heterogeneous data 
bases Is an 1in>ortant endeavor If we are to fully utilize our national ^ 
information resource, and take full advantage of the potential that the 
.ca^uter offers. Bates (8) describes no less than 29 useful search tactics for 
li^rovlng the query process. She pleads for searching decision rules that will 
minimize cognitive strain and begs for anything that will help searchers work 
faster [and better]. Thp existence of stress and strain In this process is 
confirmed by Standard (91 1 *^o fojnd the presence of "peak levels of pressure" 
In three critical sub-stages of the search process: (1) the selection of 
systems, data bases and 'search strategies, (2) evaluating Interim results 
(whife connect time Is still running), and (3) presenting results and costs to^ 
the user. Thi? finding reportedly was based on the opinion of searchers who 
collectively had performed 10,000 searches. 

. 14 
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3.0 PUtfOSE 

« ■ • ■ ■■ - ■ "' * . 

§ , ■ • , 

A sajor piirpose of thfs^ research to Investigate a particular 
approach, for .laprovtug search strategy preparation, aiid ultimately retHeval 
perforwante. for *tho$e qn-11i» searches Involving fit^ than one data has^. The 
approach was to integrate existing controlled vocabularies Into onfe-3ata.bas0 
and to exploit the en^nawis Intellectual effort repre^ted H>y such a 
coBVo'slte f41e for the purpose of 1«|Slrov1ng learch^trate^les. 
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4,0 oBJEcnfEs M score 



* ' ■ 

'The coa4>1neet objectives of the tm> research grants were to: 

-^1) Improve and expand earlier subject switching concepts 
.Knd desists \ 



(2) 



(4) 



(5) 



Test and evaluate VSS In cdhtrolled and uBcOntrolled 
envlronwiBnts 

^Detenilne the Inf lu«ice of certain * technical factors 
on VSS perforaance. Including; vocabulary size,* 
mfflber' of vocabularies, vocabulary r1n11ar1ty/d1s- 
slfflllarlty, and type of snitching, strategies e^loyed 

Oetenalne ihe 1nfla«fee of certain hi«an factors on 
VSS perforaance. Including: education and experience 
of participants, their search proficiencies In 
various subject areas, and' ^ user and Intemedlar^ 
evaluations of VSS « ♦ 

Oetemlne the extent of user satisfaction for 
searches performed with and without VSS enhancmnt 



(6) Characterize a futuristic Interaedlary-free search 
environment Involving VSS, ♦ 

The scope of work was Halted' to a systea cwkIsHm of 10 to 15 
controlled vocabular1esava11«*le lif^ »ch1ne-readable form and at a reasonable 
cost to the project. Formal testing was Halted to 7 field sites: 

A few scope aodlflcatlwis were aade to the original objectives and 
|Scope during the actual course of the research. First, a futuristic, 
intermediary-free search scenario was not perforaed because the hours budgeted 
for this effort were actually needed In bringing up a revised and modified 
VSS. However, one addltlonaV activity ««s undertaken that was not originally 
planned. An update of the previous »er survey of online searcWng behavior 
An6 patterns was con<hicted. This update was considered laijortant from the 
stam^lnt of deteralnlng the shift. If any, in users on-line searching 
patterns and preferences with tire. 

The scope of field testing was modified fot one.prliwry reason: It 
was decided to acquire data from a larger segtent of ^online users than was 
.possible under the Haltatlon of 7 field sites. To accomsiodate this scope 
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modification, 5 out of the 7 field sites where T6nfl<i^%ting ¥»as -to have 
occurred were- drwk tn . favor , of less formal evaluat^^^ involvitig a iiwcli 
. bA)ader audience, This modification resultj8<k in an ev%lu|(^|lon of VSS by 65 
• interwediarfes instead of an estimated 14. . The larg^; auiience also provided 
more insight into the problea than i«uld have been ^ possible with a sn»ller 
• audience. ' ' » 



I 
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S,0 (WERYiaiflF.lfSS (PRKENTfERSIOr) ; p * 

VSS Is a s«and-ftlon6, on-line, experimental data base containing the 
subject descriptors or naln entries and all the "syndetic relationships found 
fff c^trol led vocabularies, ttierefore, actual postings or frequency totals are 
not included ^In this data baie hikause' Its origin Is not the bibliographic 
record or conventional inverted file. ^ . * 

Curr^tlV, VSS resides on Battelle's CDC computer and uses a 
cortJinatlon of Fortran and W«IS-IR softwre (Version 4.0) fo^ data anageaent 
and retrieval. Access tp the data base. Is via the TYMNET telecomwnlcatlons 
netwrk or direct dial service. , ' . 

In this experlawtal aode, the on-line user wishing* to enhance a" 
search strategy via VSS wust fW Battelle's -computer , then, 

interrogate the VSS data base and exaain/ the output, and finally log off. 
-Any resultant search strategy, enhanced or not, wist be re-entered In the 
usual fashloff by dialing up a retrieval service and data base coi*1nation 
appropriate for the search. In a production nodfr, the VSS data base *K^ld 
s1«|)ly reside as another data base within each retrieval services* host 
conputer. and woulds run with their softwre. This would el Iwinate thq^- extra 
login step now required In the experimental model. 

Like most other on-line data bases,' VSS contains inverted ^iles, 
data base records (for display or printing), and system commands. (Ml Ike most 
other data bases, VSS •|:«ita1ns #only search terms or keywords useful for 
performir^ on-line searches in othea* data bases. 

Since the. Inverted files a'mj' data base records are created from 
existing, machine-readable controlled vocabularies, these vocabularies are 
described first, followed by the ira^erted files, the concept file (data base 
records), systea'conmands, and the switching options. 

5.1 VSS Vocabularies * • 

VSS contains all of the lead-term entries and syndetic relationships 
found. In 15 controlled vocabularies. These vocabularies (see Table 1) are 
organized into four major categories, and they are called modules, accessible 
only l>y module. No inter-module access is permitted; 

• * 

■ I ■ . 



The physical sciences vocabularies, with the exception of the. IROM^ 
Thesaurus, wre acquired under an .earlier N5F grant aiid represent late 1978 
acquisitions. The IRON Thesaurus and all other vocabularies shoii^n tn Table 1 
were acqulr^ In late 1979 and Mr ly 1980 under a separate NSF grant. For the 
fflost part, these vocabularies were ctortated to the project or acquired for a 
'noiBlnal f ee Aie to th6 research nature ol^ the work. . • 

All vocabularies were acquired -In machine readable .form- via magnetic 
iape.- However, In order to mlfllBihe the actual cost to each supplier, no 
fomat restrjctjons'were laposed upon then. This, of course^ placed a burden - 
^rt the VSS staff to- handle ^nd process 12 different "as supplied" vocabulary 

formats. ^' * - / . ' - - ' • * • 

As shown in table 1, the - smallest ^Tpdabulary Included* In 'VSS Is 

Management Content.s— the lai^es^ Is Me§H (Medical Subject- Headings) from 
National Library of -Medicine. This mix of vocabularies was assembled to study 
•switching In various fields of endeavor rather tHaf\ Ipoklry efcluslvely at any 
one • subject a^6a as wals the case Jn previous reseafch^The actual acquisi- 
tions represent the mbst readily available machine-readable vocabularies of 
the time. ' v ^ ... 

The Chemical Abstracts. Conc^t Edit Fne*.(CA) was placed In both ,the 
life and physical science categories because of Its scope and Impact with 
ot^r ■ vocabulaitfes In these categories. . Alsoi previous survey results-' 
Indicated that |*0pular multi-data base searches frequently Involved CA In 
combination with those data bases In the life and physical sciences areas of 
VSS. \ . . - 

•The INSPEC Thesaurus was an Important acquisition becauser. of the 
links (use/used, for cross refererfces) It had established between British an^ 
Ai^rlcan spellings. Thus colour and color, centre ahd center wer^ useful 
synonym linkages available only In the INSPEC Thesaurus. The IRON Thesaurus, 
•devel<H)ed by Battelle for the IKOH Information Center was useful becSause It 
was trilingual. English, Spanish and Portuguese'. - 

, Although the Biosciences Infwraatlon Service provided on lyvone file, 
the MAIF, their concept code (Cfi) seemed to be an important synohym-type ' 
structure In which certain biological concepts were also given systematic 
numeric cocles. Therefore, ' toe VSS staff * created a secohd BlOiSIS vocabulary, 
BIOSIS-C, which was an Irilbrse of the main-entry concept-code relationship 

t9 » 
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TMKJl VSS fPC^nUHUES 



iksutioiisii^ 



J. 



HmT »tat IT KT 



All 



Mp8t« Contents 
Sub Total 




9.3 
1.3 
10.$ 

0.1 
. S.9 
14.0 

16.9 
j4.1 
13.9 
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42.2 
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0.9 
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0. 9 
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43.7 


5.8 . 
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13.0 


187.1: 
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9.4 


'79.9 
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0 
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14.0 


96.1 


\ sr.2 
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10.3 


39.3 


423.8 


• 
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0 
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2.9 


19.7 


0 


69.4 


2^.1 
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0: 


1.9 


17.1 


0 


9.8 




9.3 


48.1* 
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20.5 
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6.8 


52.9 
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54.8 


84.9 


1.6 


160.8 


• vO.l 


144.5 


132.6 


27.0 


416.8 



Um RT - mUtnd tom 



ContMts - 



J* 



I Inftra T^wnis 



Contents TlMscurvs 



Abs 



fldfJ^- Ttm Master Index Autlnrlty File (NIAF) fnm BXOSIS^ In the VSS 

version of vocabttlary* BC (Biological Cades)- and CC 

(Concept Cotes) are Invalid. 
BXOSIS-C' - A special version of BXOSIS in Hhlch CC (concept cndes) are v' 

valid but the'com^epts are Invalid. 
MeSH - Ihe Pledlcal Subject Headings of tbe mmm systei used at the 

National Library of IMIcIim; 
NISH-R • A special subset of NeSN In Mhlch si^stmce nonas are Invalid 

Imt ^1r efilvalent ctaslcal registry iwatMrs are v^lld. 
MeSH-S - The Inverse of NiSi-R^ 
CA - atypical /Ibttracts Ooncapt Edit Ffie. 



DOE . 
EI 

IRSf»EC 
IRON 

NASA 



Chealcal Abstracts Concept Edit File. 

- DapartMnt of ^tergy Ittesrarus. 

. Si£ieet Headings for Ekiglneering, fraai Engineering Xnforaatlon 
faci (f^wly Engineering Index). • 

- ' INSKC TbesMims. 

- A trlllngoal theseenis on Iron aatallurgy. Lan^iages 1ne1u<te 
^llsh^^gwlsb, Md Pfl^tupMse. 



*** NilSA tape contained only UF relationships, but VSS preprocessing softmre, 
expect ase rvlatlonsblp. 
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found In the BIOSIS file. In oth6r words. In VSS the wain headings in BIOSIS 
Were 'valid and the concept codes were, invalid, while in the BIOSIS-C file the 
inverse was tru^. The d€K:1s1on to create a second file to laTrror this 
iRportant relationship wade it possible to access VSS with either the concept 
or the concept code to perform switching^ • . 

In a slailar fashion, the liational Library of Medicine furnished one 
tape but a special/ se^Jlent dn th.is tape contained MeSH headings mapped to 
Chemical Abstracts Registry *»M*ers, This tape iws used to create three- files, 
one for regular «eSH headings, one -1n*wh1ch CA R^lstry Ik^rs were valid 
(substances invalid), and onrin which substarrces were valid (Registry Nuabers 

invalid)'. ' , " ^ ' 
V It can also be seen ' f row Jab le 1 that sowe vocabularies contain 

cAain sywletic relationships which others do not. Thus, there were no 
broader /narrow terw relationships In the life sciences area. Likewise, there 
were no'^ecial scope notes in the business and social science areas. 

Finally, it should be pointed out, although it is not leparent froo 
Table 1, that the vocabularies themselves represent r miJctyre of subject; 
heading schanes and authory^ lists. ThUs, they all represwt soiae form of a 
controlled vocabulary iut theiy are not all exclusively of one type' This is 
why the system is referred to as the Vocabulary Switching Syst^ and why' we 
refer to thew as vocabularies rather than thesauri. . . 

A new feature added to VSS during this grant period (1982-a3) was 
the related terw. This relationship ,was ignored in earlier jrersions of the 
system because the browse aspect, which a related terw provides, was 
considered much less ii^jortant than the synonym, generic terw (BT/NT), and 
scope note features. Since those early days, we -have changed pur opinion 
^ about the value, and indeed the purpose, of VSS. The related term is now 
considered to be an important diwension and potentially useful search terw. In 
addition, VSS is now considered a tool to enhance search strategies not .merely 
a ^bol to identify synonyms and generically related terms. The usefulness of 
' VSS as a tool for enhancing searches Is increased by providing all the 
syndetic rela-tionships available In the original v<K:abular1es. As cah be seen 
from Table 1, the Inclusion of related terms added a considerate dimension %o 
VSS, about 193 thousand new terras altogether. 



21 



13 



VSS were: 



Battel! e processed each acquired vocabMlary so as to present the 
syndetic relationships whenever and wherever possible. The syndetic relation- 
ships' captured by VSS are: «a1n,-hdid1ng (lead tern) entries, conventional 
scope notes, «pec1al scope notes, use (and-SM) cross references, use for. 
cross references, broader and narro*«r terms, and related tertft^. 

Special -scope ^tes were cireated fqr VSS to' handle certain types^of 
synedetic relatibnshlps found in the acquired j^ocabularieS which could not bq 
easily handled any* other wi^ by our thesaurus softwaref 

The two specific relationships handled by a specif scopi note ^ 

. • ■ ■ . ' J 

. . * . y 7 

• Terai A • . •» ■ 
use either tera B or term C or . . . , ^ 

• TerfB.A " ; 
' use Term B, Jefm C, Term D ... all W ed 

* Each vocabulary Included In VSS is flssigned a unique letter vcode. 

This code is raapjred to B bit tabl« consisting of 24 jfositltJns (bne^posltion fs 
designated for each assigned code). Therefore, the maxiawm nuaber of 
vocabularies that can be handle^ in «ny; one version of VSS 'is. ^4. The 
vocabulary code preserves the original source of each »a1rf ent^ar or synd6t1c 
relationship stored and retrieved by VSS.* The* code- always displayed with 

all VSS output so that the user knows what',.controlled-Vocabulary search terjsis 

. ■ 

are available for each data base. . • 

. Each acquired vocabulary was pr«:essed via a specially written 
program into a common VSS input foipmat. (Appendix A stwws th^ cuyfent format) . 
It was from this common format that all V§S Inverted files and data base 

records wer« eventually created. * 

» 

u ■ 

5.2 Inverted Files 

VSS Irtverted files are of four types': 
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term file 

• word file; * 

• stem file 

• stem phrase file 
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These .files are created froa the teaporary file by, executing a ' 
series of coraputer prograss and a string algorithm.. See Appendix B. 

The tenB file Is an Inverted file containing every valtd and invalid, 
main entry terw listed jn-^acji vocabulary* These teras can be single or laulti- 
word' terms or phrasesl mis file is created by a piece of software *»hich 
"sanitizes" each tens and assigns a pointer (concept number parcel) for .use 
retrieval of the data. The teNi file is, keyed by tera and uses its pointer as 
a key to retrieval of actual vocabulary records froa the concept file, a 
sequential file ke^^ by pointer (corK:ept nunbier parcel). The pointer 
functions like an accession miaber — It is the k^ to retrieval of vocabulary 
records froa the concept file (In essence the jlata base records) - 

The sanitlzaiion algoritiw squeezes all special characters froa^ aain 
entries, such, as.' hyphws, parentheses,* sealcolmis, coaaas, ete. This same 
algorltha is Invoked when users enter teras at the teralnal. Thus, a jrfiole 
class^of aorphologic probleBK deal1ns| with punctuation are eliair^ed as a. 
barrier to retrieval. Users n^ not r^aei^^r if or how to punttuate f tera to 
use VSS. . ' ^ 

^ • Only rarely was it nepessi^ry . to subject a term to additional 
sin1t1?ati^ to laprwe rc^itrleval. The aost.cawon case was ttie use of special 
codes by vocabulary suppliers (e.g. COSATI category codes) . If retained as 
part of tiie aa in entry, special codes wwild re^iilre^rs to r«i!lJer thea -to 
retrieve the tera or to switch on It, or the user would have to truncal every 
term he/she was unsure, of In order to succeed. Therefore, these types of 
strings, .though uncoaaon, were reaoyed froa aain entries before their 

. Insertion into the tera file. .1 

' The word file Is an Inverted file of individual words created by 
disasseatollng every lead terajn the temporary file into its. coiBponent words, 
tt is like thfe tera in every other respect. ^ 

♦ A major utility of this file Is to provide the capability for^ 

retrieving portions of nulti-wrd ter^ or phrases. In other words. If a VSS 
user starts with a 3-word phrase, he/she can request retrieval of all other 
phrases which contain any two words used In the original phrase, or any single 
word used in the original phrase. The systerti, using an Internal scgring 
algorithm, retrieves all ^phrases which satisfy the user's requirements, 
ciefault Is lOQ percent, j 

23 . 
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The stew file is similar to- the word file except that air Individual 
word'$ are proceslsed by a steaming algorlthn to create roots or steins of the' 
words. This same steraalng algorithm Is Invoked when users specify a stem file 
search at the terminal. It sta^ their input the same wfior It stems the 
inverted file entries. YSS produces output when matches between a user's 
stemmed. Input. and a stem file entry are f (Hind. * , • 

A major use for the stem file Is to locate all vocabulary entries 
with the *^same root. TTius, coaputers, computer, coliputing, co^utatlon, 
. computational, all contain the sam^ ytem, compu, and all would be considered 
valid output if. the user specified A ^em file search* on any one of these 
words. 

t' The stem phrase file Is an Inverted file created by dlsasseiribllng 

each main entry term In the te^rary file Into its component words, applying- 
the. standing algorithms to each word, and concatenating the resulting ^tems. 
The result Is an Inverted file of stem strings w^ere the string Is cowposediof 
the stems of each -unique word In the original term. Obviously, a stem phrase 
file entry Is only creat«1 for terms or phrases containing two or wore words. 
For example, electrical • machining would be r^resented as ELECTRMACH In the 
stem phrase file. » ' * 

TKfes^purpose of this flle'ls to retrieve a variety of terms where the 
Individual wrds within the term differ ohly by ending. Thus, electrical 

f'^'machlnes, electric machining, etc., all produce the same cwicatenated stem and 
would be considered valid output. • ' ' 



5.3 > -Concept File j-. 

The iSiwt -f4 le Is a symbol 1^ keye^l file, accessed by thp concept 
nunfcer parcel. . ini1i file contains 'i^mfc can be*, considered the data bise 
records, 'albeit the r^ords are vocabulary records rather than references. 

An Individual record In- this file contains the unsanltlzed text of 
the lead term, the vocabulary bit table. Indicating the sottrce(s) of the term, 
and pointers to other terms In cohc^t file-., The other terms Include: co- 
related terms, narrower /broadetr te/ms, related terms, and the USE ten^. All 
'VSS output originates from the cdncepLf^^lle. The four Inverted files merely 



provide* the means of gaining access to the concept file.' tf course, this Is 
^11 transparent to the'VSS user. . - , , 

The" use of unsanltlzed teras was a new feature added to VSS during 
this grant period. It adds an laportant diaension to this ty^tm because ,tJfe^ 
VSS user q^n enter a tena Incorrectly punctuated artd retrieve a correctly 

punctuated term. ' • . 

Table 2 shows the relative sizes of Inverted and concept' files for 
the four major search oodiiles In VSS.. As can be seen, the final file size for 
alTfouf nodules was about 89 alllJon characters. The largest VSS file. Is the 
terra file with tHe'exceptlon of this social sciences libdule, idiere the concept 
was slightly larger. The awllest VSS- file is the stea file. The largest 
modula was Life Sciences, the siaallest was the Business Module. On the 
average, about 97 characters of storage are required to 1rn)leilieft^ one 
thesaurus entry with all Its^ relationships accordltig to the specifications of 
this version of VSSl ' " * ' ■ 

TABLE 2 VSS FILE SIZE BT fdOWftiRY SET 

(Mo. dHoricters X 106) / • ' 



File 


• 

^ Business 

« 


Life - 
Sciences 


Social 
Sciences 


n^ical 


* » 

Total 


Concept 


0.76 • 


. 9.76 


■1.39 


' 7.54 


19-45 


Term 


lAt 


10.99 


- . 'U33 


7.79 


21.23 


Stem 


0.59 


8.35 


. '0.66 * 


4.23 


13.83 


Word 


0.73 


9.14 


o.a) 


.4.88 


15.55 


Phrase 


0.93 . . 


. 9.65 


1.24 


6.99 


. 18.81 


Total 


4.13 


47.89 


5.42 


31.43 


88.87 
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5.4 ' Systea Coaaands 

* • * 

■ . . ■ ^ ' • » . • ' 

there have been no changes to the VSS system conwands since the last 
report (10). table 3 suoBsrlzes th6 Available comnds. During this renting 
period, the systei cowahds mere. used alB»st exclusively by the VSS staff to 
study the performance of VSS prior to the field elvaluatlon. Hoi»ever, during 
the field evaluation VSS iy^tm cowBands were Bade transparent to the user. 
Tills was-dohe to relieve the burden dn the user of becoming i^ sVSS systems 
expert before evaluating «1ts usefulness. 

5.5 SjfftctilBq Odtlows " ^ • . 

• . - ' * 

Switching (^tlons are logical Instructions urtilch tell VS5 what. to do 

■f , ^ IPs " 

with the user query tera and i^iat files to access (In essence what type of 
switching to perforam). They also define the types of records be retrieved and 
displayed from the VSS concept file. These opt 1 wis (Table 4) are used In 
cdnjunctlon with the systea coiwaqds to effect a retrieval In VSS. • 

Option 21 was added to VSS during the reporting *per^o<* (See Table 
4). This option produces relatM terms, if any exist, to ]the term entered by 
the VSS user. It differs from option 9 whicfi produces co-related terms. 
Related and co-related terms are different relationships 1n VSS. See Appendix 
C for a «pre detailed d1scuss4on of the aval^lable switching t^tions.-*^ ^ 

' Switching options, like System cosnands, were used by the VSS staff 
prior to the field evaluation to study various switching scenarios. -During the 
field evaluation these (^t1ons were wade transparent to the user by designing 
a menu containing six switching strategies, five of which were pre-defined. 
When a user selected one of the five pre-defined strategies, VSS aut«aat1cally 
1nvoked> the switching options defined for that choice. The ^§1xth strategy, 
user-defined, required the VSS user to supply his/her own options. 

5.6 ttelna VSS ^^^-^ 

VSS can be used in one of two modes, command or menu-driven. The 
comand mode requires user specification of what VSS must do, while the menu- 

'26 . 
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TMLE 3 AVAIUIBLE STSTDf CQMMDtl 



QOXV 



STORE 



FWCT XOW FBRyORMto 

i fy^^jf^^ all •provloos Monmds eaceopt (KJXT and STORS 
Catalpg» a' «witehia9 st^tegy ' 



OOKiZC 



I 



SBTOPS 



8BTV0C 
V0CC8T 



,ts doplieata term in the otatpat 
« . * 

Ou tp u t froB all preoedix>9 <^>erati<»i8 bec^aaes in^ut 
fdr iia«t op«raticm within a svfitching strategy 

Diaplimi vaXidi^ ^ a ten aoxoss ail Vooabalarias 
cvt^xdieas of VOCCl^ parameter ^ 

S«9^rasses le^ texss with Imtwrtrtert Bollean 
connectors (e»g* .A8D,) . 

odtput from preceding .operation hecooss input' tot * 
nut operation, witbin a sMitohing strategy 

peasits display of fcc^s notes 

Def iMs' the nnBbeir of adjacent tezas to be ' 
. d^ylay ^^ . ..^ ■ 

^nisesothe previous switching strategy 

Detendjies bow -such of an input phrase Msst be present 
in an output pbrase, if not 100%* 

Braids the previously selected vocabularies 

Defines the mavimnw noadber of teras desired f ran 

ea<rii vocabulary j^^^^ ^ ^ 



n 



I 



DBB06 
SBXZST 

8ETLST 
SfiTPIQI 



OKBUG 



Activates a trace funoti^ for system debuggSd 

Alliodro swi^diii^ to ocNesenlctt at sosn use/^ designated 
point within a switching strategy 

Allows switching to tersdnate af »mm user designa- 
ted. &aA point other than the Imt option in the 
strategy ' 

Allows alteration of ' the array ZSPRM 

Allows presMtore termination Qf a strategy if one or 
more tenas are produced ^ 

Deactivates the trace function 
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/ 




driven mode requires only a few slaple choices. The Menu-driven node was a 
f newly ImpleiBented feature during this r^ortfng period. 

The Bienu-dr1v«i Biode Is shown in Figure X. It Involves only a few 
menus plus specification of the nui*er of terns to be displayed. Fro» then on. 
the user slaply enUrs teras to' be Switched, one. at a tlfne. The ^SS systeia 
resppnds to each tens entered with a result or a oiessage tha(t switching was 
unsuccessful. Users can change their inenu choices or specification statement 
at ahy tlwe. ' 

Figure 2 Is an eiaiq^le of a typical teralnal session iising the menu- 
driven mode, in this exanple.^i user selected all six vocabularies available 
In the flfe(|sc1ences module, the browse mode of switching and 6 terms per 
vocabulary. Then the user entered a Chemtcal Abstracts Service Registry 
'Number, 90(^-97-4 for switching. The system was able to execute the brtme 
^ option and produced a table of results. The ;table shows the type of switching 
performed under jthe column TERM TYPE, the vocabulary containing the term found, 
under the column heading VOCAJ, and the term found under the Qolumn* TERM, 
Appendix Obtains a more detailed explanation of the results and also how fo 
operate VSS In the commnd mode. v. ' t 

If switching Is unsuccess^l, the system s1fl<)ly Issue^ a message to 
that effect. The user is free to fiiiter another term or change the menu choices 
.and re-enter the sasie term." t 
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TMLE4 AVAOMLE SUItGHIlK QPnOIS 

V . - ' ...... 



'code 


— -y, ^ . ' ' 

* ■ ' • • r 

, Switching Cation 


, • ... 

C^ti<»i NicMnaae 


< 

• 

• 

Type of outimt Produced 


3 


TERM FILE ACCES8 - ACCEPT LEM? TB»B'GNLSr 


Bxftct Matching 

* * • 


Exact Matohlx^ 


♦ 


TimH FXLi ACCESS ACCETT StT^TCBABLB TBBNS OKLSf • 


Partial SynonyB Svltchll^ 




5 . 


TEiM PILE ACCESS - ACCEPT UBAD -1- »fITCHAaiS TE»ffi 

• . • 




" Bsact Na^tches Plus SynonyBS 


6 


STiai PHRASE FILE - ACCEPT LEAD SWITCHABIf TSBNS 

; • - . . 


. StoBi Phr^e SHltcdilng 


l^asA Variants Based on 
StflBBiin9 


i 


WORD FILE ACCESS - (SBIPCT)" IffiTCH SEQOIREE) 


• • 

«brd File Sifltchlng 


Phrase 'Variants Based on Wor^ 


8 


STEM FIL& ACCESS (OTPCT) MASqB VBS^OtSm 


Ward Stenlng 


Word Variants 


9 


OCmCBST Vim ACCESS - ACCEPT RELATED TERMS GMLST 


Oo-Related lierm SNfltchlug . 


Co-Related Terns 


10 " 


CONCEPT FILE ACCESS - ACCEPT HARBOH TERMS OSUi 


Rarroner S^rai SMtcbln^^ 


Generic Terae 

y 


11 


CONCEPT FILE ACCESS - ACCEPT BEGAD TERNS OHUf 


Broader Tersi Snitching 


Geneicic Terns 


^2 


TERM FII£ ADC^ - LIST (^SSfmXJ) ADJACENT TEMIS 


Adjacency Siflt^Mng 


Adjapent Texas 


13 


WORD FILE ACCESS - LIST (2*SE13UX7) AOJACEHT NdOlS 




AdjcKsent Moulds 


14 


STai FILE ACCESS - LIST (2*SETADJ) ^ADJACENT PHRAS]^ 


• 


Adjac«tt Steaus 


-15 


STEM PHRASE FILE - LIST' (2*SETADJ) ADJACENT PHRASES 




Adjacent: Phr^es 




ACCEPT I£AD 4- USE -t- DCOBLE OSS T^MS 






17 


ACCEPT LEAD + USE ■¥ tXmLE USE + USE-FOR TERMS 




Syncmyss * \ ^ 


18 


ACCEPT LEAD -1- * OOOBLS C6E -i- BSE FOR -f 
. DOUBLE l^ED-«PR TEI9I . 






19 


ACCEPT LEAD MULTIPLE USE -t- MULTIPIf USED-FOR 

• 


Exhaustive Syixxym Sid.tdhln9 




20 


ACCEPT LEAD ■¥ ISILTIPLB USE -1- MOLTIPLE DSSI>-FCm -l-^ 
RELATED TEIttiS , 1 

CONCEPT FILE ACCESS - ACCEPT &SLATBD TEBMS ONLY ' 




^Sif&onyas PXi^ C^&elated 

Tem 

• 
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^ 5 


Selated Teni Switching 


Related Terms' — ^ 
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FlfiUHEl NEW-DRIVa NODE'OF fSS 



CHOOSE 
VOCABULARY 

SET ' 



Biwkiass 



CHOOSE 
^WITCHING 
''STRATEQY 



lift Sctencs 

\ 

\ 

3-«-NaiT0w«r 



<3 .ami 4) 
6 — Utar DeftaMd 



SPSaPY THS MAXUMURft MUMB&I jp^ TERMS 
. TO BE DiSnj^YEO m VOCAWLAR Y. 
EVfTER A MUMBER. 



i 



T 



BNTJBR 
YOUR TERM 

01^ 
STIV^TiOY 



I 



RESULTS^ 
OR « 
SWITCH NOT 
SUCCESSFUL 
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UElCtt«,TO *YSS* - VOQ^UCARY SWITCHXJIG StSTEM 
USI.lt BATTEUE'S (MTA l\fmsmsn SYSTEM, .BASIS 



vss ctmTAiisr Fotm vocabi^y sets 
1^ m^mss 

' A. AtfX 6. immmn cotirENTs 

2* SOCIAL SClEtlCE 

A. ERIC B. PSYCH ABSTRACTS 
U LIFE SCIEHCE 

A. BIOSIS B. CA ' 

€• VSSH " 
4- PHYSICAL SCIEHCE* 

A, m. B. CA ^ 

C. EI 0. IHSrtC 

E. IROri F.. NASA 



nmm z vss tdwiml session in menu 



PLEAK,$ELECT I OF THE ^ VOCABULARY SETS ^ 
BY EflTERING EmCK 1, 2, 3, OR 4 
?3 

RE<H£STEO FILE SET IS 0M4ME - COimwS ^ 

PLEASE SELECT THE VOCABULARIES OF lifTBIEST . 
BY ENTERII® T)€ LEITER(S) SEPARATED BY COmAS; 
FOR mf«>LE: A,B,C 

3-\lf E SCIEICE . . . 

A. BXOSI^ B. BIOSI^-C 

C. CA -<. 
U HESH-R 



*.» 
* . 



* ^. MESH 
F. HE»(-S 



VSS PROVIDES ¥m S SUITCHI^ FEATIAES: 
. 1. SYfWtmtS ^ 

2- B WUSE • 

3- NARIUMSl TERMS 

4- BROADER TERtIS 

5- NAiSlOICR/BimAOER TERMS 
B- OTHERCUSER-OiFIW) 

PLEASE SaECT 1 OF TIC 6 OPTIt^j; 
BY EHTEglMG EITHER 1* .2, 3. 4. S» OR S 

72 ^* 

SPECIFY. THE MAXimJIt ?nHm OF TER»S ' 
TO «E DISPLAYED PER VOCABULARY. ^ 
ENTER A NUtBER. 

?6 * J 

i , 

PLEASE EUTER SEARCH TERM ffit VSS CO^IANO 
. '79005-97-4 



SMITCH SUCCESSFUL 
TERK TYPE YOCAB 



TERM 



YOUR TERM 
SYMOWYH ♦ 
SYMOWW ♦ 
ADJ-LEAO 
AOJ-LEAO 
AOJ-LEAO 
AOJ-LEAO 
ADJ-LEAO 
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HESH-R 9005-97-4 

ilESH USE nODOPROTEIHS P^EIN/*ANALQGS ORED 

iCSH-S PROTAHOHE 

MESH-R '9004-94-8 

KESH-R 9004-99-3 

HESH-R 9005-32-7 . . 

l«SH-R 9(»5-66-7 ^ 32 

!SSH-R 9005-67-8 
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6.0 VSSr RESEARCH ACnVltlES 



Research, proceeded aiding two fronts; software development and system 
evaluation.^ 

» ♦ ■ . 

... " * * 

*' ,*. m ^ . 

6.1 Software Oevelopweiit ^ ^ ■ , 

ftjr4ng the course of work pn ^ Grant I^r-791U90L (1979-81), VSS 
software was converted over to a new release of iBie data aahagement software, 
and xm runs under BASIS 4.0. Also, ten new jfocabulartes were acquired for * 
VSS, necessitating the rebuilding of all VSS flWS. 

During fe-ant 1ST-8U1497 (19^-83). a completely new user Interface 
was written for VSS to maktf It easier to use. the system ami Interpret output.^ 
This was done In anticipation of a large-scale field test with no user 
training. Also', the related term relationship found 1n many of the 15 
vocabularies was ♦dded to the VSS files, and the. concept file was redesigned 
to handle .this new relationship. Finally, all sanitized terms In the concept 
ffle^^terms without >»ec1el characters) were r^aced with ufls^nltlzed ter^ 
to more accurately repres«it the search terms actually used by the data, bases 
accessible via VSS vocabularies.* . ' . 

6.2 System Evaluetlon 

Several typ€| of evaluatlws We'conAicted with VSS: 

• fon^T evaluatlomof selected switching istrategles 

e> formal evaluation involving «id users. Intermediaries, 
VSS, and publicly available- data* bases 

* " • 

e Informal evaluation Involving Information brokers, 
librarians. Information science and library school 
graduate students, and data base vendor staff 

In addition, the 197? On-line Users Survey was updated. Each of 
these evaluation efforts ami thfe on-line suryey Is described separately In 
subsequent sections of this r«i»ort. Each section contjalns a description of the 
test methodology and the results obtained. * 
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i.o afLmim OF SMrromc stratoies ^ 

Uith. the addition of ten new vocabularies (a total- of 15 unique 
vocabularies altogether) and with tlie clustering of all vocabularies Into one 
of fbur.aajor sbdules by si*ject tiAtter, It was felt -thfct a new ^t of tera- 
^level analyses, «3lianar to those comhicted earlier (10), was needed. These 
analyses help deteralne the perfdtmance of various, switching strategies. 

The objective of this evaluation ms to detertilne the perf onaance of 
several switches strategies with the goal df utfliz'ing wly the best ones In 

. foriaal and Infonaal field evaluations of the-syste«. A further objective was 
to conpare the performance of each aajor nodule- to determine the effects, if 
any, of subject area, vocabulary size, and imber of vocabularies on switching 

. perfonaance. * ^ * . 

7.1 itettiodology 

m ■ . ... , » ^ 

' A"«SWltchtng strategy consists of one or more swItcMng options {see 
Table 4) and zero to several systes comnds (Table 3). These options and 
cooaands are set-up by the user In a specified sequential order for execution 
by Vss. Each strategy to be analyzed was miabered to facilitate reporting. 
. . Seven switching strategies w«re defined and evaluated within each of 
'the^four VSS oodules. Hoiever, tN life sciences awdule was broken down Into 
^ three aodules. small, nedlua and large. In order to stM^bWthe eff«:t of aodule 
size on switching perforaance. Therefore, 42 unique analyse \[«re 'performed (7 
strategies x4 BK)du1es). . 

The seven switching strategies consisted of the following switching 

options and systea coiinamis: ^ 



Strategy 
Wuitoer 

1 
2 
3 
4 
5 
6 
7 



Type of 
StraJ 



browse 

bnmse 

broader 

narrower 

synonym 

browse 

browse 



Switching Qptlois 
(in 9Fder of exeoitlwi) 

3,4.19,g,6J0,7,8,l6 

3,4,6,M»20 

3,4,19,11 

3,4,10.19 > 

3V4.19,0,9 

3,4,6,7,8,19.9 

8,19 



System Co—inds 

sc(^, setDCt««50, fulV 
same as '(1) • 
scope, fullu 
sc<^, expand, fullv 
same as 
same as 
same as 
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Each strategy, was designed to test different combinations of switching, 

options. / • 

Some strategies invoked relatively sWie switching. VFor ^xai^le, 
strategy 7 Involved only word stews and synonyw. Other strateglesNpre father^ 
coBplex and Included aany features available In. VSS. 

The analyses were perforaed by passing a serlected set of search 
teras through each of the sevw pre-defined strategies and comparing VSS 
output to the ir^t. Four different sets of search terms were used as Input, 
one tailored for each of the four saJpr nodules In VSS: business, social 
sciences, life sciences, and physical sciences. 

These ten* sets wer* derived froa actual queries submitted to 
searchers who routinely provide online retrieval services. The business terms 
were provided by » the Public Library of Coluaibus and Franklin County (PLCFC); 
social science terms were obtained from the Mechanized Information Center (The 
Ohio State University) ami FLCFC; life science terms were provided by the 
Itealth Sciences Library (The Ohio State Ifcilverslty); * and physical * science 
terns »«re acquired from the Battelle Library. Appendix D shows each of the 
four term sets used^ for the analyses. Since output was voluminous, the term 
sets were randomly sample to estimate performance for each of the 42 unique 
coBfclnations mentioned earlier. These randraly selected terms are Identified 
in, the term sets shown In Appendix 0. ^ 

Switching strategy performance was estimated ^ by determining the 
amount of relevant VSS output compared to the Input search term and the •amount 
of CPU time rmcessary to perform a particular, switching strategy. 

The analyses were ^widucted in the following manne?: 

(1) An entire term set was used as input for each of the . 
seven strategies, ♦ for the appropriate VSS module 
(e.g. buslri^ss terms were used as input to seven 
^witching strategies' in the business moduTe, life 
Sfrience terms became input to the "same • severr 
strategies for three' life science modules, etc). 

(2) All VSS output was 'savecl for subsequepf analysis. 
Each of the four ir^^ut -term sets was raflBomly sailed 
to estimate switching performance for each set. A 10 

. percent sanple was taken from tfie physical science 
term set and a 20 percent sample was taken from each 
of the three remaining sets. 
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Relevance jiK^eaents were mde bye the project staff 
fwall output relative to the -li^Mt terms In the 
r^ectlve saiqiles. This provided an estimate df the 
quality of each switching strategy. Quantity of 
output for each ' strategy i^as also cosplled. An 
overall perforauice valtre vas ^ proihict of quality 
and (^ahtlty. 




(4i CPU t1a» was compiled ^or each analysis and Included 
as a perforsance aeasure. 

* * 

A typical exawpl'e of the type of analysis perforaed Is shpwn In the 
exaRple below: • 



' ExB^ile 
TBM-LEVa iMALYSIS 



TERM 

DIALYSIS « 



1 



NEXT TERM 



VSS ' 
OUTPUT 

DIALYSIS 
CC 64712 
DIALYSANCE ' 
DIALYSATE 

rCXT OUTPUT 



TOTAL OUTPUT 



RBLEVARCE 
BECISIOWS 

relevant ~ 
retevant 
nonnrelevant' 
non-relevant 

• 

BXMre declsimi 



I 



TOTAL RELEVANCE 



^ relevant 
output 

XX T^evant 
outputs- 
overall 
% relevant 



\ It -aust be po1nt«l "out that #:e an output result was judged 
relevant or non-relevant» that dwHslon was consistently a«?11ed across* all 
sev^ siiltchlng strategies, which produced that 'identical output. In othlJ* 
^ words. If DIALYSATE was^ Judged Irrelevant for switching strategy 1 In the life 
science aodule. then by conv^tlon It was irrelevant If strategies 2 through 7 
also produced this ou^ut where DIALYSIS was the 1iH»ut term. 
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7.2 Results 



. A composite of all 42 analyses Is shown In Table 5. This table lists, 
the overall perforsance value deteNilned for each Individual analysis as wefl 
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as the overall raiding of strategies and VSS modules. TTje overall performance 
value was the product of output quantity and estimated quality based on 
relevance. For example, the performance value of 75 for switching strategy 1 
within the VSS business oodule represents the product of an output calculated 
at 893t and a relevance estimate of 8« (89 X 84 " 75).' 

It can be seen fro« this table that switching strategies 1, 2, and 6 
produced* consistently higher perfSroance scores across the board than any of . 
the other four strategies. Tlie^e thr^ strategies scored particularly hlgJt In 
the business and social sciences modules which can be explained by the fact 
that these strategies were among the most roljdst. The fact that strategy 2 
performed sLIghtly better than strategy 1 In each module Is undoubtedly chie to 
the fact that it excluded some features which tend Uo produce non-relevant 
output/ Including such features as stoBlng and adjacency (H)t1ons. 

Several factors contributed to the low ranking 6f strategies 3 >nd 
4. First, broader/narrower terms were not available Jn business or life ^ 
sciences modules, hence, figures In Table B for these modules reflect only the 
synonym switching taking place, ftowever. In both the social sciences and 

phj^slcal sclertces modules where broader/narrower terms were available. 

performance values 'were still the. lowest of all seven strategies^ this can 
partially be explained by a second factor, the difficulty of making a 
relevance decision for broader and narrower terms In the abs«K:e of the end 
user. Of all the syndetic relationships available, the hierarchical one Is .the 
iiSost dependent on an end-user Judgaaent. In .this' study, we erred on the side 
of non-relevance when there was any dwjbt. 

UwHcIng at VSS modules. It can be seen. that the social science 
module produced consistently higher scores than all other VSS modules for all 
but two switching strategies (1 and 2). Only the business module outperformed 
the social science mdiJules for strategies 1 and 2. Likewise, the business 
module pf^uced consistently higher scores than either the life sconce or 
physical science modules. ^ 
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TABLE 5 rarammicE of vss nodules 



Life Sciences* - 



1 


75** 


68 


28 


23 


23 


39 


2- 


2 .' 


76 


68 


36 


25 


25 


40 


1. 


3 


40 


60 


15 


14 


, 10 


29 


6 


4 


40 


57 


15 


14 


10 


24 


7 


5 


51 


67 


30 , 


.18 


19* 


35 


4 


6 


^54 


73 


31 


23 


23 


36 


3 


1 


44 


65 


.25 


18 


19 


28 


5 


Module Ranking 


2 


1. 


4 


5 


6 


3 





*si»n fflodule - BIOSIS. BIOSIS-C, MeSH « ' eu c 

laedltflR «odule - BIOSIS. BIOSIS-C. HeSI. Meai-R* MeSH-S 
large aodule - BIOSIS, BIOSIS-C, HeSH, MeSH-R, HeSH-S, CA 

**the perforwnce weasure Is tip pro*ict of VSS output (In**) and 
_ e ^tlaiited relevance of output (In %) 



These resuHs suggest that the sore slallar the vocabulafles with 
respect to s^Ject «tt«r :aBd «fwlet1c C^strw^ the better they «rtl) 
furwtlon as switching vocabularies. This result Is what one would expect. In 
this research, the two social «:1ence vocabularies i*ere slallaf In subject^ 
Batter and syndetic constructs; the business vodtfwilarles were very slallar !n 
subj^t natter but differed caisliter^ly In syndetic structure. Both the life 
sc1efK:e and physical science vocabularies differed In subject matter and 

structure. . 

An unexpected result was the Inverse relationship between nodule 
size and perfornance. In this particular case, the "»1V life science nodule 
perforwed consistently higher than the "large" life scleircfe^ule across the 
seven switching strategies. He suspect that at least ti>o factors contributed 
to this anowalous result: (1) the vocabularies becane - Increasingly dlfffcujt 
to evaluate with Increasing nodule size because of the abstract nature of 
BIOSIS concept codes and CA registry nunbers (we erred conservatlveljr **ifen 
unsure i^out a code w registry nuii*>er), and (2) an apparently higher degree 
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of Inconpatlbirity among these vocabularies, than originally thought, both 
from content and syndetic viewpoints.. 

Tables 6 ti^ugh XI siwiaarlze the perfonaances for each of the VSS 
flKidules studied. Column ^ In each table shoifs .the aaxltrnw possible coverage. 
This tiurriber represents the nu^^r of tergis submitted for pos^ble switching 
multiplied by the nurter of target vocabularies specified multiplied *by the 
•vocabulary count specified. In other iwrds, if two terms A and ,8, are 
submitted for switching across two vocabularies, Vi and Va, and a-. vocabulary 
count of two Is wcified (meaning the user wllf accept MP to two switched 
outputs per vocabulary), the maximum possible coverage, if switching were 100 
percent efficient, is 8 (2x2x2). , . 

For^is analysis, the maximum possible coverage was different fof 
each module,' according to the schedule below: 



Nodule 


Nuriier of 
Terms 


Nuri^ of 
VocaiNilaries 


Vocabulary 
Coimt 


Coverage 


• 

Business 


143 


t 


1 


286 


Social Science 


113 


Z 


1 


' 226 


Life Science 


222 


3-6 


1 


666-1332 


Physical Science 


666 


6 ' 


1 


3996 



Column B, in Tables 6 to 11, records actual coverage, Jthat'is t^e 
actual amount of output gwierated by VSS for a particular strategy. Column C 
shows the percent coverage [{A i B) X lod] achieved by a particular switching! 
Coverage ranged from 98 percent to 19 percent, with the social science module 
exhibiting the best coverage percentages among the four modules. 

Columns D and E record the results of relevance assessments made on 
all the VSS output generated by the sampled inpMt, Sample size was 20 percent 
of the input term sets for business, social and life sciences, or 28, 22, and 
44 terms respectively. For physical soiencesr the saiiple size was 10 percent, 
or 66 terms. Relevant output from these sampled input terms was recorded in 
column D, non-relevant terms in column E. 
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Coluan F shows the estlwated pfercerttage of relevant output for each 
strategy/^ule coablnatlon. Die range was 100 to 40 percent ^relevance. 

Adjusted coverage* colunn Is the product of percent coverage 
(coluan C) and percent rel€!vance (coliwn F) and was used as' the bencf»ark for 
coiparlng strategies and ondules (Table 5). . ' , 

IHe final colw. H» shows th^ cuaulatlve Cfn; X\m to execytite a 
comlete set of tenas through the noaul^l In other words. It took 20.5 CPU 
seconds to perfora' switching on 143' busli^ess terns In the business nodule for 
strategy 1, or an average of 0.14 secomTs j»er Ir^ut tern; It can be seen 
that broader/narrower tern -switching (strategies 3 and 4) were the nost 
efficient while browse switching wW steawlr^ and phrase parsing occerrlng 
^arly In the strategy ms roi^hly IW ilnes slwer, and therefore, extrenely 
Inefficient. 

Table 12 sinnarlzes the detailed perfomance of each switching 
strata by VSS vqcabulary nochile. In these tables the progressive effect of 
each switching option can be observed for ^ch strategy, and cowparlsons of 
Indlvfidual options across the various nodules are readjly nnde. 

It can be seen that exact natch lBg> .that Is,' a character-by- 
character conparlson of users' terns with li^erted file terns, contributed a 
significant portion of the actual output .produced by VSS. Of course, exact 
natchii^ is nothing nore than an indication of ttie subject overlap anon^ 
controlled vocabularies. Ignoring honogi^y. Matujrally this result Is a 
constant across all switching strategies because the vocabularies remained 
constant throughout the reporting period. ' . 

In the case of the business and social sciem:es nodules, 'exact 
natching prodtjced the greatest output, 37 and 49 percent respectively, ijfi all 
strategies test^ exc^t for strategy 7 -*^ich did not incorporate this opttipn. , 
This result indicates the degree to which these vocabularies are coopatible 
compared to life and physical sciences. ... 

Both life and physical science nodules pHxIuced considerably less 
output with tt« exact natching option than either the business or social 
science nodules, indicating a tendency toward subject disslnilarity. 

The word qption in VSS was the next best producer of VSS- output in 
strategies where it was used wHh the exception of strategy 6. This, of 
course, says nothing about the quality (relevance) of output produced by this 
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TMU6 ?wmm£ gp B^ifiEss wmjE 





Total 
Possible 


Coverage 


(C) 
< 


(D) 

( 


1^) 

m 


(F) 

• ■%' 
R 


(6) 
./MJusted 


(H) • 
Ctn 
Tlee 
(Sec) 


Dyne 1 




254 

1 


• 89 


47 


9 ' 


84 


75 ' 


20.5 / 






241 

* 


84 


45 


• 

S 


90 


76 


19.2 


BUDS 3 * 


286 


115 


40 


30 


0 


100 

• 


40 . 


^ '0.* . 


BWDS 4 


286 ' 


/ lis 


^ 40 


30 


0 


100^ 


' 40 


* 0.9 


BUDS 5 


286 


.146 • 


51 


> 




loq 


51*' 


3.7 


BUDS 6 


1 

286 




59 - 

t 


30' 


3 


91 


54 


99.3 


BUDS 7 


286 


167 




24 


8 


.75 


44 


13.7 



TABLE? PEXimiAiRX OF SOCIilL SCIORS POm 



(A) 
Total 



Coverage 
Jf S Possible 



Meal 



% 



m (E) (F) 



No- No* 
R HI 



% 
R 



(8) • 
/UJusted 



(H) 
Cub 
Tioe 
(Sec) 



PUDS 1. 
PUDS 2 
PUDS 3 
PUDS 4 
PUDS 5 
PUDS 6 
PUDS 7 



226 

« 

226 
226 
226 
226 



^26 
226 



221 
216 
135 
127 
^142 
168 
167 



98 
96 

57 
67 
75 
74 



32 13 

26 .0. 



25 
24 
3p 
21 



0 
0 
1 
3 



69 

71 
100 
100 
100 

97 
'88 



68 
68 
60 

57 
67 
73 
6S 



^ 



19.3 
18.0 
1.5 
4.1 

p.'s 

144.1 
42.0 



R » maoxit of relevant output based on random saispllng 

NR » anount of non-relevant output based on random sampling 

Cun time » cuiiiu.lat1ve tPU time for entire strategy 
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TABLES PaU"flRN(WCC OT SMftlL tIFE SCICTCE MODULE . 





Total 
^ CO¥0rMe 
Possible 


(B) 

ActueV 
,CDVfi*8ge 


(C) 

• 

Csverage 


(0) 
R 


(E) 
MR 


(F) 

% 

R 


(6) 

mam _ _ ^ _ ■ • 

Adjusted 
Coverige 
OtF 


•(H) 
Cub. 
Tine 
(5ec) 




. 666 




68 


69 


101 


41 


28 


65 


NWDS 2 


666 

♦ 


415 . 


^ 62 


44' 


.32 


58 


36 


*64 


1WDS 3 


; 666 


173 . 


26 


29. 


22 


57 


15 


2.7 


NMOS 4 ^^ 666 


173 


26 


29 


22 


57 


15 


3.1 




4 


m 

m 


•i3 


^ 


4 


91 


I 30 


9.8 


MfDS 6 


666 


m • 


38 


41 


9 


82 


31 


217 

f 


HMDS T 


666 


251 . 


38 


33 




67 


• 25 

• 


57 




4 


m 

WMCE OF MB 


)tm LIFE SCARCE 


NOOULE 






(») 

Total 
Possible 


w 


(C) 
% 

Cdwuge 


(D) 
R 


NR 


(F) 

% 

R 

r 


(S) 
Adjusted 

OiF 


(H) 
On. 
Tine 
(Sec) 


Olios 1 


1110 


• 

623 


56 ^ 


65 


93 


41 


23 


203 


(WDS 2 


1110 ; 


546 


49 


54 


49 


52 


25 


199 


OWDS 3 


1110 


207 


19 


35 


11 


76 


14 


6 


OWDS 4 


1110 


207 


19 


35 


11 


76 


14 


• 

6 


OWI^ 5 


1110 


275 ^ 


25 


44 


16 


73 


18 


16 


OWDS 6 


1110 


368 


33 


SO 


21 


70 


23 


492 


OWDS 7 


1110 


341 ' 


30 


37 


29 


^ 59 • 


18 


83 
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• 
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TABLE 10 PEMmWKE OF LARS LIFE SCIENCE tmSJE 





(A) 
ToUl 
Coverage 
Possible 


(B) 

. Actsal 
Coveih^ 


(C) 
% 

Coverage 


CD) 

* 

R 


(E) 
NR 


(F) 

X 

R 


(6) 
Adjusted 

Coverage 
CxF 


(H) 

Tine 
(Sec) 


mus 1 


1332 


7^ 


58 


66 


98 


' « 

40 


23 


91 


MWDS 2 


1332 


669 


50 


56 


56 


50 


25 


89 


MMDS 3 


1332 


253 

• 


19 


34 


32 


52 


10 


U 


MUDS 4 


1332 


253 


19. 

» 


34 


32 


52 


10 


11 


MUDS 5 


1332 


^ 339 


25 


45 


14 


76 


19 


15 


MUDS 6 


1332 


* 439 


33 


52 


22 


70 


23 


517 


MUDS 7 


.> 1332 

• 


• 

403 


30 


43 


26 


62 


19 

* 


86 

• 
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' <A) 


(BJ 


(C) 


(D) 


(E) 




(6) 


(H) 




ToUl 




* 








Adjusted 




It 


Coveram 


B ' Acteel 


% 


• 




% 




Tiae 


Possible Coverage Co 


verage 


R 




R 




ISec) 


^S 1 


1(3996 


■k33l8 


83 


123 


139 


47 


" 39 


203 


SUDS 2 


3996 


> 3073" • 


78 


105 

* • 


102 


51 


40 


450 


SUDS 3 


3996 


1157 


» 


66 


3 


96 


29 


20 


SUDS 4 


3996 


1075 


28 * 


52 


8 


87 


24 


33 


SUDS 5 


3996 


1944 


49 


90 


35 


72 


35" 


39 


SUDS 6 


3996 


2645 


66 


^9 


82 


55 


36 


5140 


'SUDS 7 


39% 


2525 


63 




85 


45 


28 
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optfon. Itoifever, It can b€ observed that the wor^ t^tlon produced consistently 
high amounts of output ^across all VSS nodules (Table 12). To fully understand 
this result. It Is 1sp<M*tant to understand the word option In VSS. 

* The «ord option breaks down^ a user.'s term Into Its conponent words 
and searches the Inverted word file with thea. A scoring algorltha Internal to 
VSS applies a user-st^pl led threshold value to 'word file hits before 
displaying ^ny VSS outpjut. The default threshold Is 100 percent. For ex^le. 
If 'the It^ut term Is fWCLEiW PWER; and the default of 100 Is accfeptable, then 
all VSS ouliut flMst have 100 percent of the Input ^raCs) present. Thus, 
HUCLEM POWgr' PI:AIIT SITINSS satisfies the #fault. Slrallarly, If a synonyw 
relationship exists^ where the Inval Id -«^ry Iwitalns 100 percent of the Input 
tena, then the word option In VSS Is satlsf led'and the valid term Is displayed 
as output. The word cation In VSJ can be a source of prolific output 
because the threshold can be set lower than 100 percent by the. user. Also, 
terms longer than the ^Ir^ut ter« will satisfy a default of 100 percent 
provided the VSS terw contains all of the Ir^ut'terw. * ♦ 

For exa«ple. If a user enters ESTHOSEH In the life science nodule, 
there are «any terw within the F}b§H, ifeSH-R ^arkl MeSH-S vocabularies 
conUlnIng ^1s tern JESTROGEH SULFOTRAHSFERASE,' ESTRDSEK 2-HYDROXYLASE, 
ESTROGEN AMALOK to mention a few) which could satisfy the word option. Thus, * 
the word option tends to produce voluan. output, 'e^>ec1ally. at lower threshold 
values and In cases where si wUin-wrd terns are entered. Thresholds Of 50 and* 
100 percent were used for stiWteglei 1 and 6, respectively. 

Vocabulary switching behajre^ In a 'nanner virtually Identical to data 
base retrieval, theory, that IsL^ the higher the recall, the lower, the 
precision* The pr1rK:1pTe difference be^tween VSS strategies 2 and 6 Is the 
output producec^ via the word optW {apUm\ 20 Is In reality options* 19 and ? 

coiflb.1ned>. ' . * « 

Strategy 2 r^resents high coverage (recall) because a low ifehreshold 
(50*) for word and stea options was specified while strategy 6 r^resents high 
relevance (precision) because a high threshold (lOOX) for word and sten 
options was eavlo^. These two strategies behave as one would . expect— high 
recall, low pneclslon and vice versa. 

The related phrase option produced output equal to or exceeding t)ie 
various VSS synonyw options. Also, It is observed that this option produced a 
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TABLE 12 DETAILED PERFDRMRCE OF IMiniDIIAL SNnCHm SnUTEGIES 



SMltdilna Strategy 
(1) 



Mtchmg 
C|»t1oiis 



Type of 
Mtaing 




Life Sciences 



NBdlue 



is 




3 
4 

19 
9 
6 

10 
7 
8 

12 



exact mtch 

synonym 

synmym 

co-related 

related phrase 

narrower 

word 

stm 

adjacency 



37 
2 
1 
0 

11 
0 

28 
5 
5 



49 
4 

3 
0 

11 
0 

27 
2 
2 



19 
6 
"1 
0 
7 
0 

27 
2 
6 



12 
4 
3 
1 
4 
0 

22 
4 



13 
3 
3 
1 
5 
0 

23 
3 
7 



Total Output (In %) 
Estimated Relevance {in %) ^ 
overall Strategy Perfomiahce 
(InJCf 



89 
84 

75 



98 
69 
68 



68 
41 
28 



56 

41 
23 



58 
40 
23 



19 
5 
1 
0 

23 
1 

24 
5 
5 



83 
47 
39 



SirltchliM Strategy 



3*^ 

4 

6 

7 
8 
20 



exact natch 
synonym 

related phrase 

word 

stem 

symHiyms & 
correlate! 



37 

2 

11 
29 
•5 
0 



49 
•4 

12 
29 
2 
0 



19 
6 
7 

27 
2 
1 



12 
4 
5 

23 
4 

, 4 



13 
3 
5 

25 
4 

1 



Total Output (In %) 
Estimated Relevance (In %) 
Overall Strategy Performance 
(in %T 



84 
90 
76 



68 



62 
58 
36 



49 

52 
25 



50 

50 
25 



19 
5 

23 
26 
5 
0 



78 
51 
.40 



/ 
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TMU 12 . (Continued) 



SWltdilng Strategy 



Sifltdilng 
Onions 



Mtcblng 




111 



Life SciencMHi 



NadlUB 




3 exact antch 

4 ' synonyms 
19 synonyos 

11 broader term 


37 
2 

*1 
0 


.49 
A 

3 

4^ 


19 

6 ' 


12 
4 
3 
0 


13 

3. 

3 

0 


^ • 

s 

1 ^ 

5 


Total Output 

EstlRfoted Relevance (In %) 
Overall Strateoy P^rfdnaance 
(1>? t) 


40 
100 
40 


60 
100 
60 


is 

57 
15 


19 
76 
* 14 

* * 


. 19 
52 
10 


30 
96 
29 


smtdiliM strategy 


* 












t 

3 exact iMtch 

4 syifionyns 

10 narroMer terns 
19 synonyns 


37 
2 
0 

. 1 


49 
4 

1 
3 


19 
6 
0 
1 


12 

4 ^ 

0'^ 
3 '. 


13 
3 
0 
3 


• 

19 
5 
3 
1 


Total Output (In 
Estimated Relevance (In %) 
Overall Strategy Perforpance 
(Ini) 


40 
100 
40 


57 
100 
57> 


26 
57 
15 


19 
76 

. 14 


19 
52 
10 


28 • 

87 

24 


* 

Switching Strategy 
15) . 














- 3 exact match 
4 synonym 
19 synonj/m 
16 related phrase > 
9' co-related 


37 
2 

'■ 1 

n 

0 


49 
4 

3 
11 
0 


19 
6 

. 1 
. 7 
0 




12 
4 
3 
4 

2 


-13 
3 
3 
5 
1 


19 
■« 5 

1 
23 

1 / 


Total Output {in %] ^ 
Estimated Relevance (In X) 
Overall Strategy Performance 
(infi - 


51 


67 
100 
67 


33 
91 
30 


25 
73 
18 


25 
76 
19 


49 * 

72 
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TMLE 12 (Continued) 



SiHtchlna Strategy 



Swltdilng 
Q|it1aiis 



Tsipeof 
Swltdiliig 




NBdllM 




3 
4 

7 



V related i^rase 
wnrd' 
stem 

co-related 



37 

2 

lit 
6 
2 
1 
0 



49 
4 

11 
7 
3 

'1 
0 



19 
6 
7 
3 
2 



1 

0 . 



12 
4- 
5 
8 
2 
2 
0 



13 
3 



2 
2 
1 



19 
5 

24 
14 
4 
0 
0 



Total Output iint) 
Estlisated Relevance (In %) 
Overall Strategy Perforsance 

(In I) • 



59 
91 
54 



75 
97 
73 



. 38 
82 
31 



33 
70 
23 



.33 
70 
23 



66 
55 
36 



Sirltdiino Strategy 

IT) < i 



8 
19 



related phrase 
synonym 



1 



73 
1 



36 
1 



28 
2 



28 
2 



62 
1 



Total Output (In «) 
Estlnated Relevance (In %) 
Overall Strategy PerforBance 
• (In I) 



59 
75 
44 



74 
65 



37 
67 
25 



59 
18 



62 
19 



63 
45 
28 



47 
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consistent amount of output regardless of Its position in a strategy (Table 
12. Strategies X, 2, 5 ^m! 6). Strategy 7 (Table < 12) featured the related 
phrase option by positioning It first and excluding virtually all otN'er 
(^tions. except a synon>n option. Strategy 7 conf inns what one would expe<^t of 
• the related phras^ option, nmly high relevance. ; 

f' Synonyns proved to be less significant than originally thought. 
ATthough VSS has a powerful synonyn switching capability^ less than 8 percent 
of all switched output can be attributed to these options. Option 4, a sj^nonyn 
option, accounted for 2 to 6 percent while (H)tion 19 contributed another 0 to 
3 percent. 

* One explanation for the Idwer-ithan-expected perfomiance oj the 
synonym option is the specificity of synonym teras themselves within 
. individual vocabularies* ^The more ' unique a ■USE" or "USED FOR" cross 
reference, the less likely it will be found elsewhere— hence, no switching, 
y For example, consider the entry: 

Cosmic ray showers and bursts* 

UF Auger showers ' 



r 



cascade showers 
cosmic ray jets • 
EAS ♦ 
extensive air showers 
showers, cosmic ray 



Even though six synonyn» or quasi -synonyms are given for the entry, "cokmic 
ray showers and bursts", the probability that any one of them will be found in 
another vocabulary in VSS is very small. . • - - 

Another explapation for lower than expected synonym switching is the 
fewer number of vocabularies per module in this veQion of VSS. Our earlier 
work with ten integrated energy vocabularies indicated that Option 4 alone was 
capable of switching at the rate of about 10 percent (11). Options 9 and 20, 
which also prodiKe syfKmyms, were not available back then. Therefore, we 
conclude that the synergy observed earlier was due to the large nuni>er of 



♦Entry taken from INSPEC Thesaurus 

ERJC 



; 

7 



ERIC 



19 . ' 

• » 

vocabularies per aodule, and that a slnllar synergy would be observed In the 
current version of VSS If It too had 10 or aore vocabularies per iaodu]e. 

Also, In this analysis a vocabulary count of one was specified for 
tte. seven strategies evaluated! This set-up Limited' output to one. term per 
vocabulary. Had a' higher vocabulary count been specified, more synonyws woald 

have been produced. 

ltetther-*f«adeMor narrower tern options produced mich VSS -outpnt 
(Table 12, Strategy 1, 3 .and^ 4) In the apdules which contained such 
relationships (social physical sciences) J However, C(»is1der1ng ^e 

conditions under which these options were analyzed, the" result was not too 
surprising. The conditions were as follows: . 

(1) The vocabulary count was set at one (1) 

(2) If an exact watchr or synonym was 1d«rt1f1ed In a 
particular vocabulary, the vocabulary count for that i 
vocabulary was already satisfied, by definition 

« 

(3) Only If entered terss had broader or narrawEr ter«s 
could VSS atteapt to locate such ter«s {switch) In, 
other vocabularies where exact satch or synonym 
options had failed. ^ ^ 

The adjacwy option (strategy 1) contributed ooderateTy toward the 
overall covera^ In various aodules; holder, the relevance of adjacent tirws 

was low.' - ^ ' * . 

In suawary. the terai-l0vel analyses suggested that strategies 1 and 
2 would perform better than any of the others test«J for this particular 
version of VSS. Performance also was noticeably better In the business and 
social science modules than In life or physical science modules, due primarily 
to a higher degree of siAject overlap ^ong the business and social science 

vocabularies. . ^ 

As a result of this analysis, a "browse strategy", was designed for 
eventual ' use in the field evaluation .of VSS. That browse strategy combined 
most of the options used^ln strategies 1 and 2, biit it excluded broader and 
narrower term opVfUns and Iritluded our new option, related terms. The biKtuse 
strategy that was eventually field tested Is shown In AppencHx C. Users mre 
f encouraged to set the vocabulary count to a high number (5 to 10) to produce 

♦ 
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40 

aaxtiWia output for each pptlpn within the strategy. In a sense* high-recall 
switching was encoU)raged because experience showed that the best use of VSS Is 
as a "shopping list". Users can handle relatively long tists of teras and 
quiqkly decide which ones are valuable for eventual search strategy enhance- 
nents or liodlf Icatlons. <^ 



1 

\ 
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8.0 BP USER ETOBgHTS' • ^ 

Several md user tests :iiere conducted 4rt order to evaluate ' VSS In 
actual cw-llrfe reference retrieval situations. Xh^purpose of the tests was to 
coopare bibliographic citations pro&iced by ^S^lf led searches with those 
i^rodsced under norsal condltlwis. 

One series of tests utilized a tafides or Iterative design, that* Is, 
the original 'search strategy was aodlfled only after , the original c^ery had 
been negotiated, and the on-line search perfprtKd. The test Involved one. end 
user and one Interaedlary. 

A second so'les of tests Involved a parallel or slmultan^s design.- 
Here, VSS was Introduced at the start of the user- <^ery and Involved two 
Interaedlarles for eacli end user. Each Intoraedlary p«^on8ed a separate 
search for the saae fnd user, howler, one search Involved VSS *<i11e the other 
was conducted in the u^al Bnnner. 



S 



8-1 ttetHodology 



The tandea. or 'iteifttive test wis conducted at a site designated as 
Test Site 1. It was set »^ in the following i«y: 

Subsequent to an end^r query and successful coapletion 
of an off-line' search session, the Interwediary Involved 
noted »*iether the search just coapleted was a possible 
candidate for the VSS test. To be a candidate, the 
ca^leted search' had to Involve data bases i^ich VSS could 
handle via one of its 14 vocabularies. Wnen a candidate . 
search was id«»tified,' the ewl user was contacted and 

^ asked to participatiB in the test.. Participation Meant that 
the user's ofiginal c^iery would be nodifled and , 
resub«itted. The user was asked to evaluate all witput 
. froa both searches for relevance. The second search was 
done*- at no cost to the «id user. Search teras used in the 
original on-lirw search strategy were entered into VSS for i 
possible switching.' Then the- intersediary evaluated VSS 
output and nodlf led the original search strategy If <te«Med 
appropriate to do so.. Next, the aodlfled search was rerun 
against those data bases named in the ^original search 

* Sich aatchfd up with VSS vocabularies. Duplicate 
citations were eliainated beti«en the two searches within 



51 



42' 



each unique file (data Jwise). Users received tow ^separate 
outputs for evaluation: citations proAiced by the original ^ . 
search and those produced by a VSS-iaodlf led search. ♦ "^^ 

• # 

The parallel or -sliailtaneous test nas contoted at tuo different 
sites at two different tiaes (abou^2 years apart) using tifo slallar but not 
Identical proceAires. Consec^ently, the results are r«port«l separately for 
each test site, **1ch *re referred to as Test 51te 2 and Test Site 3. The test 
was set up In the following way: 

< 

Both parellel tests ai|»10)fM tN saae basic «(p«rlB»ital 
design as shown In Table 13. Two Intecaedlarles were 
Involved In each search beginning ifltlf the end-user 
presearch Interview* *(M the o<M nuabered- searches. 
1nterBwt!«ry A used VSS to develi^ tiie search strategy, 
while Intwwedlary B <»mfeicted tl« smm swrdt In his/her 
usual Manner unassisted by VSS. On even numred se«rches, 
the Interaedlarles reversed their roles. This design was 
eii|>loyed to neutralize any skill differences maong the 
Interaedlarles so that aggregate results for a test site 
> would represwt an objective coaparlson of VSS versus non- 

VSS s^rches. 



r 



TMU13 O mUB tt fL KSIWfW THE 

rmniSL TEST imLniB vss 



End Ita- 


Sevcb 


lateraedlafy A 


..imenKdlanr 8 


1 


r 


VSS 


• 

non-VSS 


2 


2 


non-VSS 


VSS 


3 


3 


' VSS 


non-VSS 


4 


4 


non-VSS 


VSS 



The manner In which this design. was lepleoented varied at 
each site because an attempt ms mde to leprove the 
procedure^ for the third site based on the feedback froa 
the second site. 

Appendix E contains o^ies of tN fdras and proce<fejres / 

used at each site. Si^ara^ evaluation fores were desired 

for the Interwedlary and the end user. Both sites used a 

written set of procedures^ Intenaedlarles at both sites 

were given 1 to 2 hour training sessions In the use of 

VSS. Procedures were carefully explained and reviewed. 

ERIC . > V 52 
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Interwedlafles were encouriged to practice with the JgS 
system for sone tlae Nfore cowwicing tiie experiment ^ 
.eitil users. 

The atjor differences In experimental, methodology bet>«en 
.the tmo sites ifere as foll^; 



0 



&*eater interaction between intermediary and ^d *user 
was permitted for Test Site 3 exper1m«its than for Tejt. 
Site 2 experiments. This change, .was In 
coBBlalnts raised by Test Site 2 participants (See Step 
5 oTprocechire in Appendix E). . , . 

different procetkires were outlined for Test Site, 3 to 
isolate YSS c1tat1«is from nw-VSS citations ;^and 
citations sommon to both (cof|»are steps 8 and 9 In Test 
S1^ 2 with the same st^s In Test Site 3). 



Forms 



• A different ^d user evaluation form was designed^ for 
Test Site 3 to match the modified procedures used. at 
Vest Site 3 for 1<^t1fying clUtlon sets. Also, by 
having the ewl user complete a performarwe t^le on 
h1s/»Sr ww*sheet it was felt that tl^ participant 
would have an appreciation for (^lestlons Ht>iest1ons 7 
and 8) dealing with satisfaction of various sets of 
citations.' 

Finally, different visions of.VSS were used at §ach test 
site. Test Site 2 used an earlier, more cumbei-saae version 
of VS5, lAlch required the Intermediary to Sf t t^p his/her 
om switching strategy and to deal wItt s««w**t cryptic' 
YSS output. Related termi were not available with this 
earlier version. 

All of these conditions were different for Test Site 3, 
that Is, swishing strategies were predefined, VSS output 
avoided cryptic coding, arKJ related ter^ were available. 
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8.2 Rcsalts 

« * * 

A total of eight searches Mas corKkicted' at three test sites over the 
course of the t«o grants. Table l4 suaparlzes the resMlts for all test sites. 

Two searches w«re conchKted at Test Stte 1. One sear<:ti Involved the 
"extraction of flavor frOB fnHI^ Ju1ces"» the otT^r was a search for Hi^-. 
cQRuonent acrylic i^slves". T^les 15 and 16 show tiielmm^ and YSS search 
strategies aiid results for these two <pier1es. ^ • 

, The searches dlffef^ greatly In aibject «tter, c^Juplexity and^ 
f^mgib«r cltktions^iiirlfite^ off-line. However, ^ daU bases searched were 
- |lBlTar lit both caies, Aid user satlsfactlw wIth/tN VSS ^ertlon of t<|e 
.. output ifaS yeryffavor^le in cases. In fatt, Raiser «ho requested 
. InfbnBa'tlon on adheslves thoi^l^t the JTSS'im^ of tN output^wis better than 
the mm-VSS portion, alth^gii a ifef M< » m reason could not be given for tlie 
perceived d1ff«re«:e i^lltyw The person requesting ^nforaatlon on fruit 
juices was sloply happy to have tiM extra relevant citations that a YSS- 
TOdlfted s^rch was able to produce. It, can be' observed that precision 
sufTin&d\^n virtually all the VSJf searches; towever tkis perforaaiwe factor 
did not sees to affect e^ user satisfaction with the final rd$Tt$- - 

As would be dcpected, VSS proikiced fewer relevant citations in 
nearly all searches e»c^t the CA search for aiilieslves. (At thl^ point no 
explanation can be of f ered f or the extrao«11nafy perforwince of ,VSS on the 
search for -acrylic adheslves In the CA file.) The reason ttat YSS produced 
< fewer relevant c1tat1<^s Is that It represented the new citations »*i1ch the 
* non-VSS searches failed to prmhtce. In a s^ise, what VSS accowplIsM tn this 
test was- to Idprove.thel recall of the non-YSS search. 

AhB parallel ^ slwiltaneous test was cojiducted at Test Sites 2 and 
3. A toUl of six seai4:iies was conAicted, two at Test Site 2 and four at Test 
Site 3. Tables 17 and is show the detailed results of searches conducted at 
Site 2 and Tables 19 thrdiigh 22 show the results of searches at Site 3. 

^ It can, be see^ that VSS's perfontance at Test Site 2 was l«<er 
coafwred to Noh-YSS perflprteiwe for search question - (B) and virtually Jf»n- 
. existent for search question (A). However, end user satisfaction was 
sowewhatsurprlslng, FIrU of all. the end user associated with question (A), 
Table 17, gave four evaluations as expected, one for^^ch search caid)1nat1on 
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Rerforaed (ERIC-Ncm-YSS^, ERIC-.VSS; PSYCH^-¥SS. PSYCH-VSS). The end user who 
siAaiUted question (B), Table 18,. evalueted only on tfie basis af VSS and Mon- 
VSS--an obvious coBBunlcatlon breakdoMn sotaeMhere. 

, • Second, "the 4id user wto subaltted question <A) gave ratings i^lch, 
1n effect/ averaged/ 2*5 on a 4 point scale for both Mon-VSS and VSS searches 
across the data >ases. In oth^ words, there was little difference In »e 
user's Blind between these tw M the ratings given by this end 

user s^ to belle the relevance decisions aade. With so few relevant 
docuaients In either the Hon-VSS set or the VSS set. It seens that the rating 
;ts higher than deserved^ * 

This Is caitrasted Vith search question (ft). Site 2 where the user * 
gave^nen-ySS searches if rating 'pf - 3 and VSS searches a ratli^. of 2 on the -sane 
4-point scale. Here, the ^ratings seea low In relj|t1rfn to the reliance 
decisions nade, especially fw tlie. VSS citation setv 

• The net result 1$ that one end user. (Site 2, search A) gave VSS a 
2.5 rating based on a perfonMiwe of one Relevant doci«wnt cwt of 295 while 
another «id user (Site 2 search B) gave VSS a rating of 2.0 on a performance 
of 14 relevant documents out of 

The Interaedlarles- In this tesjt rewarlced that VSS took too long and 
Tieeded aore data bases* Also, ^ey felt that test procedures were too 
structured and tM tAst ^^Inated tiielr Intewctlon witii the end user. They 
adtailtte<h that these (^r1e» Were cooplex, ^at th^y do not nonaally conduct 
awltlple data base searcHes, and that free-text searchli^ mis relied on 
heavily (aaxlM recall — re<^ed precision). Based on these cotaaent^. test 
procedures and systeia features were' ii»d1f1ed prior to the cowiefic^iBent of 

testing at Site 3. * f 

At Test Site 3, 'the relevance declslpn^ wade Yor the retrieval set 
labeled "cortoon" must be, a'dcted to both the non-VSS and VSS retrieval sets to 
arrive at co^aKable results r^ort^ for Sites 1^ and 2 because "common" 
represented :,those citations m that would have appeared In either set under 
norma V search conditions. ./ 

• It appears that users at this site were.no wore cons1st«it at rating* 

satlsfactlcm relative to the relevance. decision than the' users at Site 2. For 
\ exaiaple, the user who subnltted question 8301 rated Mon-VSS and VSS search^ 
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the sm, 3 on 5 - point scaT€, even though the VSS set contained Jiore 
Irrelevant citations than the fton>-SSS search (adding tiie cfl«Bon set to both). 
In search 8302, VSS received a higher rating than the Mon-VSS search for no 
apparent reason. Searches 8303 and __8304 seened to represent a good match 
betneen a user's raiting ana the relevance decisions; hoiieverSlnterffledlary 
perforaance was Inconsistent, particularly 1»t. search 8304. 

Although * ainor search strategy differences wre expected In the 
parallel des1||n, search 8304 r^resents a case lihere th^ VSS and Non-VSS 
strategies differed greatly. Bo^ strategies addressed, the concept of 
tacklflers In a slallar fashion but tite^ concept of adheslves ^ treated quite 
differently (s^ Table 22). The resulV wis three very d1fferentA:1tat1on sets. 
The Iton-VSS search achieved a hl^er perforaance than the VSS search because 
the concept of* adhes1v€fs was trtiated coapr^slvely In the4(on-VSS search but 
was virtually l^red In the VSS search. In a sense, this search wa^ a 
comparison of .Intermediaries rather than the system. 

Uh&i «id users at Test Site 3 were Wed to' select only Ohe 
coAlnatlon of citation ^ts, either sets A plus B or sets B plus C, where A 
was the VSS. citations, »B was the coaaon. cltatlwis, and C wais the Mon-VSS 
citations, the A plus B co«b1nat1<m was chosen 'in 3 out of 4 searches (8301, 
8302, and 8303) while the B plus C' coi*1nat1on was chos^ Iff only one search 
(8304). Again, end user evaluations were not,; always cmisistent with the 
results. For exa«ple, the correct response In search 8301 shwild have been B 
plus C. . ^ 

FIniUy, end users In .searches 8^1, 8302, and 5303 Indicated that 
they would pay nothing extra for the combination selected, while the end user 
in 8304 indicated that the combination selected would have been worth an 
additional $15. It so happens that the coab^ation chosen in 8304 was the Non- 
VSS search. This coi*inat,1on, l|on-VSS plus cowaon, yielded 82 percent- relevant 
output (42 ♦ 52 X 100) while the VSS plus common coitol nation yielded only 67 
percent relevant output (38 ♦ 57 x 100). The difference in this case was worth 
4«re to this end user. ' . 
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On the other hand, the end user in search 8303 Indicated that the 
combination chosen, a VSS search with 5 relevant citations, was mrth no 
additional noney» even though the Non-VSS search yielded 0 relevant ciltatidns. 
We can only conclude that value ^ruly Is ^n the eye of the beholder. 

On balance, VSS held Its own In these "experliaents: At Test Site 1, 
user satisfaction with VSS was high; at Site 2. fton-VSS searches came out 
slightly ahead of VSS on the user satisfaction scale; and, at Site 3, VSS 
searches came out slightly ahead. > 
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' 9.0 CTAUWnOW OF fSS Wf mTEWCPIARIES 

A field evaluation of VSSms planned and condu<^ted betiteen Narc^l4 
and Nay 31,- 1^ to ^llclt response to ithe systeia froa the Infopnatlon 
coanunlty* pnlhar.lly Ipifoimtlon brotcers. librarians. InforsBtlon and library 
scl^e students, and faculty. The evaluation placed few restrictions on 
participants other than-u^lng the systeia during an assigned period and 
coiapleting an evaluation fpi 

The objective was til» obtain user reaction to VSS in a f ield setting, 
that i|, their own work env^lrohoent with Tittle or n|||^ Interaction froa 
Battel li. The research teaii interested In obtaining data and subjective 
evaluation both at the detailed \ levels that Is, individual switching 
transactions, and over-all. . \ S 

9,1 tUtlwdoloflr 

A list of potential participants was developed frdi several sources. 
Including "Fee Based Inforoat Ion. Services" by Naranjian and Boss (12). 
Participation was strictly voluntary and all responses were coded to iaintain 
, individual and corporate ^anonyslty. Each organization c(mtacted w^ provided 
as many usemaoes and passwords as desired. 

Each partlclpailt was given a packet cmtalning an Instruction^ 
booklet (Appendix C), evaluation fona, return envel<^, and tine schedule 
(usually a one-week test period), plus two free hours of coa^niter connect tiiae 
to participate in the evaluation. 

The evaluation fora developed by the research teaia consisted of five 
I distinct areas: (1) deaographic inforaatie^n; (2) a data base profici&icy 
scale; (3) questions <teal1np with individual searches using VSS; (4) questions 
dealing wifh overall reaction to VSS; and (5) open coaaents. A sanple fom is 
shown in Appendix F. 

Questionnaire data were analyzed to obtain* the following infor> 



aation: 



t Characterization of ^e study participants 

f Correlations between demographics of the participant 
and the VSS overall evaluations 
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Correlations between VSS vocabulary aoidule (»r switching 
strategy used and the search results 

Correlations between VSS perforaance'^ ratings and such 
factors ass (A) prof1c1«icy of participant with a. VSS 
vocabulary laodule, and (B) differences «aong the 
participants thteelves as Individuals • 



e Insights froa op«)-ei^ questions about the systea. 

9.2 toults 

9.2.1 Cheracterliatloii flf tte Study Participants - 

A total of 65 participants 4took .part In $he evaluation (Tab)e 23)^ 
About 46 percent were esployed In the private sector, 28 percent .In «ie 
goVei2^t sector, 23 percent .In «^de»1a, and 3 percwt In' the "other" 
category* /M>out 43 percent of all participants were librarians. 28 per|«»t 
were eaployed In so«e capacity for on-line vendors/data base pro<^ers. 15 
perc«it were engaged In sow a^t of an Inforaatlwi/llbrary science school, 
and 14 pen^eit were Infonwtlon brokers- t 

Forty4lght of the participants had at least a Masters I^ree, and 
five had PhO's, Forty-«ie had at least ten years experience In 
11brary/1nfora»t1on activity while 40 had five or wwe years experience with 
cm-line systens.* The averaj^. years 'of exper1«Ke in HSrary/lnfomatloa 
activities* was 13*09 with a wdlan of 12.3 years whlle^' the average years of 
experience with onj^n* systeas ims 6.27 wltti a laetflan of S.44 years. . 

Participants In- the study «iere a^ed to rate their owi proficiency 
with the varlwis data bases whose vocabularies were included In tljjs version 
of VSS. table 24u shows the distribution of these self-assigned proficiency 
ratings on a data base by data base basis. 

It can be seen that participants. In general, tended to rate 
themselves as "average" or as having "little or no" proficiency. Participants 
rated themselves highest on social science data jbases (ERIC, PSYCH Abstracts) 
af¥l lowest on MASA and DOE Recim data bases. Tlie profic1»icy rating wa» used 
later on to study the effect between this variable and the participant's 
.rating of VSS performance Indlv'idual search teras^ 
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C Bt EMPLOVMENT SECTOR 



A, Private it^^) . 

B. LlbrtrVms a • . * 

. C. On-Hrte V«>dor/ 12 . 

ti^ Ulwltns. .V 12V - . r^ 



• 3. Acadailc (H-ISV - - 

■• .. . ." ■■■*■ f 

Ubrarlifis . 5 
V B. Llbrary/InfonBatlon 10 , . 
• * .r * Schools or Dtepts.- - .* • . 

A Other . ' , . ' ; 

• ' *Jlifli>ef Of ito^lclpants • ; ' 

4 ' . ■ . . . ■ . . . 

" . * ' \ ' ■ ' ' ^ : - ■ • 

9>2.2 Correlatloii Betweeii Pai^lclPMit iI iMo«raBti1c« ftrara^ Evaluation 



V 



* ftte question of Injkerest In the st*4y i«s>hether* t^^^ 
of VSS isTa fuiwtiofi Of the participant •$ experlewe or «hicat1on. Quest ions 
4 t^tnK/ the evaluation fona' deaH-i^lth experience M educatldn levels of 
the participants i^lle questions 12: thru 14 were' concerned with a i^fcrtlcl- 
pant's subjective perc«pt4<m 6f the perfonaance* Of VSS/ Pearson .corriBlatlo^^ 
iiere ccwputed between edueatlon or experience and the perforaance* ratings 

glv^. • ■■■ *^ ■ " 

Our results Indicated that the ease , of learning; using of urrief- 

sUnding VSS (Question. 12), and a 'participant's confidence (Questionaa) with 
VSS were not dojendeijt on their ^erlwft or echioation. Howeven when ask«J 
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whether YSS would- make their job easier (question 14), the. result^ Indicated 
that participants witii fewer years experience felt that VSS would nake their 
jobs easier than did wore experienced participants (table 25 A and B). This 
result was not siirprlslwg sjnce experienced 'pfirtlclpants would have a tendency 
to think of theaselves as having develop6d efficient Search methods and wwld 
be lessr Itkely tp alter thea than/|essT5xper1enced participants. 

interestingly, those^irtlclpants .with higher degrees also tended to 
Indicate that VSliS^'Wld sake Lthelr jobs e^ler. Perhaps tiiose with higher 
degrees also represented les^ experienced ufters, as wight be expected, and 
trujp/elatlonship was Msked by tffe presence of confwjnding factors. The data 
were not analyz«i for' the effects of two slrtultineoqs factors. 

Tabulation of the Individual responses to qOestlons 12 through 14 
(Figure 3) Indicate the followlr^g: ^ - 

f About 90 percent of the participants felt that VSS was 
easy-to-moderately easy to learn (Figure 3 A) 

i About 79 percent felt that VSS was easy-to-raoderately 
easy to use (Figure 3 B) 



About 75 percent felt that VSS was easy-to-4K)derately 
easy to understand (Figure 3 C) . 

ScHnewhat lower percentages of participants, 56 percent 
and 68 percent felt confident wlth^VSS's capabilities 
and output, respectively (Figure 3 D and E) • » 

About 40 percent felt the use of VSS would fnake th^it^ 
job easier, while only 9 percent felt It would sake 
their job harder (Figure 3 F) 



Jhe governmeBt sector gave VSS Its poorest rating: only 11 percent 
felt that, VSS would ^<nake their jobs easier (Table 26); Dfi the other hand, 
about 50 petcent of all parffclpants from each of the three other sectors 
(private,: atademlc, and "other*) felt that VSS would make their jobs easier. 
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Tm£ 2S USER EXmiEMCE VERSUS »PACT 
OF VSS « A SEMOet'S JOB 
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tmtC yf VSS ON JOB 
iSY BKKjOlfllffT SECTOR 



Sector 


VSS would flMke your job of 
mrchlf^ BuTtlple data bases: 


■ ■$ 

' ■ ' i 


Private 

Aca<teAlc 
Other - 


14 11 3 
2 13 3 

Jl 8 ,6 0 
1 ' 1 0 



. VSS a1 lowed eadi |>art1c1pant a choice of six switching sjtrategles. 
ind four vocabulary oodules. The dijectlve here was to evaluate VSS strategies 
and oodules against several performnce aeasufes* corresponding to: 

e Whether the VSS results were usable , ' 

e How VSS cQiqiared to one's own effort without VSS 

e the nuober of VSS terns that were usable 

e The subjective rating of IfSS for any given search 
■ (. - • ' : ■ ■ ■ • 

collectively, the part1c1|»ants atteqtted switching on 623 search 
terms. Usable results were obtained In $2 percent of the attempts (353 usable, 
%E21 unusable* 45 no response) see Figure 4A. In 48 percent of all attempts VSS 
was perceived to be >not as go«J" as what a user's own effort wouldjiave been 
for the saae search terd, while! about 26 percent thought VSS was 'better" tiian 
i^t their own effort wiHild have been (Figure 48). Tho distribution of usable 
output Is shown In Figure 4C. Table' 27 iw^sents a bre^l^down of the nuB*>er.of 
usable VSS ten^ by switching strategy. Strategy 6 produced, the greatest 
percentage o.f usable terws (92 percent), followed by Strategy 7 (85 percent) 
and Strategy 2 (65 percent). 
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*Refer to Appendix C for definition of each 

, strategy. 
**Y5S searches 



tABLE 28 VS^ RAHIIBS BY PUW OF 
.QinFIIT TEmS BY UnU ; 
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An l^rtant perforaance weasure i«s the user's' subjective rating of 
VSS, particularly for those searches «ihere usable VSS output was Indicated. 
Table 28 shoMSi^ as would be expected, that VSS ratings Increase with Increasing 
anounts of ^usable output, (^ite respectable ratings are achieved when usable 
output reaches 6-10 t^nas per ^arch request. The overall ratings for VSS were^ 
3.08 (307 usable searches); 2.35 (593 total searches). 

■ ft * f , 

9.2,3 CorrelatloBS Betweeii YSS Hrfonanee and Pscr Proficiency or tear 
IndlvlAiallty 

ft 

' While education and experience tend to be Indicators of one's 
evaluation of VS§, other factors were of 1nterest;-*An analysis was performed 
to identify the 1^ortarK:e of other factors?, including: 

• Participant differencesVe.g. differences due to over- 
all experience and education level 

' • Participant's proficiency w1^ the particular modules 
used in the search 



i^ tl 
ar^a 



• ^todule choice for a particular^arch 

The coinjuter progran BMDP3Y (Oixon, 1981) was used to perform mixed- 
model analysis of covariance on the data collected on individual searches. 
Each -participant was asked to rate VSS's performance for each search attempt- 
ed. A five-point rating scale "was used with 1 Indicating VSS was of no help 
and 5 indicating VSS was very helpful. The analysis expressed the rating as a 
function of the factors pireviously mentioned. ^ y 

The fofl owing statistical model was used in the analysis: 

^ Mu)-' H ♦ >H ♦ ?j ♦ has) ^ * • 

^^^^^ en tut feth wrcH ptrfoM by P»ft1c1pt»t j 
V iritb the m IMiUft 

m dMtM tHt tftet Am to tlM aM of i par- 
ticular Mdala* 



pa tftMtM tto tf^t ta tte miqat ^lltlaa 
I , af tHa part1c1P«»t (a.f . ^naral aiipartaaeaft 

* atfBt M atetatlon ItiwD* 

Skim ^«»tas tha nao-^yitMatlc (YMdOT) affa^s 
^ ta tHa »1<^ charactaHstICi of tha po»^ 
tlcttlar saarciit 

Pion^ dmtm$ tfia affact of tha partlelp^t*! pro- 
flclaficy^M tha ^rtlcylar »odu1a usad. 

V. 0 cpBstMtft to dttenrisad by tlw statistical 
Q MwlytU 
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this, statistical nodel assiaes that the VSS*rat1ng for each search 
Is systeaatlcally related to several factors, **h11e also being randcwly 
affected by several others. For exaaple, the effects of vocabulary nodule aind 
pantlclpant proficiency are assumed to syste8»t1cally affect the rating. The 
nodel assusKs that tte average rating assigned to Business i»dule searches (In 
general) B*y be different froa SoclaV Sciences, Life Science, or Physical 
Sciences searches. The aodel also assiaaes that VSS ratings change at a linear 
rate with participant proficiency with the particular aodulfe used In^the 
search. * . » ' 

The effects due to a participant's or a search's ui^lque 
characteristics are assuiaed to be randooi. That Is, the partlc^lpants and 
searches are assimi to be randonly selected from a larger conceptual 
population of possible participants and searches. Thus, th^haracterl sties of 
A particular search or participant that cause a rating ^o be ht^h or low 
cannot be systeaatlcally pT*ed1ct«l. Rather, their effect Is slnply to Increase 
the uncertainty In pMlctIng a rating. • - . ' 

The stAtlstlcal laodeV p^j^vldes an .apprtMch for estloiitlng thie 
magnitude of the effects due to the different, factors. The results Indicated 
that all the factors were 'statistically significant., Hoi«ver, the "estlnated 
effects of the factors differed. 

The average rating for all VSS searches, was 2.25, The effect of a 
participant's aochile choice on the VSS rating Is shown ^n Table 29. TKfs Uble 
presents the estimated average fating for each vocabulary module. The table 
Indicates that the estimated^ average rating (over all possible' participants 
and searches) for a Physical Science module search Is 2.45. However, thU 
average Increases to 3.03 wb^ only the usable searches are considered. 

Tlius, In general, V^S's performance was best for physical -science-, 
type searches while business ind 1 If -science-type searches received the lowest 
ratings. However, module choice was not a major factor InflugfKiIng the outcome 
I of VSS performance ratings. 

A participant's self -assessed .prof iclency (with a given module) was 
found to be inversely related with VSS ratings. The more proficient the 
participant (In the use of a particular module) the lower the VSa ratings 

9 

tended to be. Participants who claimed to be experts on ar given hwdule are, 
estimated* on the average, ta give ratings 0.65 points lower (on a five point 
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scale) than those participants clalalng to have no proficiency. Specifically, 
participants' *<ho rated th«selves "experts" for a particular module r? ted VSS 
1.97 on the average, Mhlle those Mho'claliaed to have no proficiency rated VSS 
2.62 on the average. 

One of the nost Inportant factors was differences aoiong Indl^^ldual 
participants. In other words, how Is the VSS perforaarwe rating affedted by 
^he uniqueness of an Individual? The estimated variability (standard 
deviation) in ratings due to differences Mong the participants was 0.63. 
Thus, a rating Is estimated to vary by as much as + 1.25 due to differences In 



jmL£ 29 ESnmTED AVERAGE RAHIKS FDR 
VSS SEAROCS BY NOOULE 



VSS Nodule 


User Ratings*, 


All Usable 
Searches Searches 

^- 


Physical Science 
Social Science 
Life Science 
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2.fS 3.03 
2.38 ' 2.90 
2.12 2.82 
2.05 2.89 


*5-Po1nt scale ^ • 



participants. In other words, if the average gating (^er all hypothetical 
participants) is 2,25, a ranijmly selected pa^icipant could give a rating as 
* low rfS l.(K), while another could give a ratii^' as high as 3.50. 

These factors I'did not explain til the variability In ratings; a 
considerable iuBOunt of jhating variation /ias unexplained. In other words, a 
considerable amount^ variation was due to uncontrolled factors that differed 
from search to search (e.g. difficulty of sealxh). The size of this variation 
could cause the ratings to vary by as faich as + 2.tK) points on the five point 
scale. In siawary, the relative importance of the factors analyzed (In 
d«:reas1ng order) are: uncontrolled search-to-search diiVerences, differences 
among participants, differences in* a partfcipant's proficiency, and differ- 
ences in the modules. 



9.2,4 Swa ry t^n fluestlons^ 

There Mere four open-endcHd questions on the VSS evaluation form 
(questions 14 through 1^). ^pendW^S llsti the verbatla responses of. each, 
participant to each question. 

. On the question of whether VSS would sake a searcher's jd>^eas1er or 
not (Question 14). participants who responded "easier" (about 40 percent) felt 
that it would save tipe ami nmey because they would not have to consult 
printed thesauri as auch* or at all. Also, they seeaed to like the Idea of 
juxtaposition of tens? froa various thesailrl and they felt that they would 
have a better Idea of what teras to use prior to searching any data. base. 

Participants i4io responded "no dlffei^e" to question 14 (about 51 
percent) felt that VSS was tiae consiaing, of little help with free text 
searches, not needed since BR$'s and Dialog's File 411 #ere sufficient 
aids, not useful enough bcmauW of too few vocabularlel, and fk)t needed If a 
seai^cher conducts a good presearch , Interview, and knows his* vocabularies. 

J Those who responded liarder" to question 14 (about 9 percent) felt 
VSS required a suj»stant1al aaount of tiae. woulil add to tr» cost of the- search 
and would, be 'another systea to learn. Soae felt that V^S needed aqre of the • 
Intelligence available In printed thesauri (e.g.. scope notes), an^l/^r felt 
the system needed ac^ work. \ v 

When asked about their overall Veactlon to .^VSS (Question 15) 
{pinions* varied widely froa "very favorable" to "not at all lapressed",. 

The three aost widely held (pinions were: 

XD VSS was ver| valuable to valuable (about 20 percent) 

(2) VSS was useful or Interest^ig bit needs ''aore work 
(about 25 percent) . >, 

(3) . VSS was cwwbersoae. ted^us or fruftratlng (about 20 
.\ pefcent) , . 

' ' ' . ' • ' ' ' :^ 

Other -pinions ificTuded Indifferent or uncomnltted. Increasingly negative, -not 

too useful, and W wortlPtt (n^s work). , * V ^ 

The prlrtclpal unolaciylng cause Qf'iis&c. frustration was the d^islon 

to provide a a^nndrlven approach to VSS with no;. option to provide' a^aore 



y" . direct access to the systes.' This was m understAndable Reaction in v1ei« of 
the rather surprising fact that rtany * participants used a 300 baud tenalnal. 
MwU'^r1y«j[^^>^ are not suited |;o slowr terwliials. , Also, 

participants %ad^ an average of about six years experience with on-line systems 
and *«re cowfbrlible with direct-search approaches. • 

Several causes appear to .be ' rela^ to the user {perception that t^e 
system needs more work { (a) vocabularies must be the Utest available and kepf 
.«.up-to-date In such ^ system, (b) vocabularies themselves should have more 
hieraiyhlcal and/or "synonym relationships to Ib?) we performance, an*^ (c) VSS 
^ Itself nee*ds new menus direct search^ and«4mpfoved switching algorltiws. 

On the*guest1pn of urtiether subject switching. In general. Is a good 
Idea (Quest 1«n 16), participants riBSponded.very^poslttvely.^ 
■» . ^iMients (Question li) tended to^reiterHe relies to earlier 

^ -questions, hmiever some nere .new. person thought there was too much noise 
* In the output, while ahother thought, the: stemming algorithm went 'too far. ^ 
. Several other points, were o^e; (1) thesauri do not stand-alone veVy well, (2). 
. title and abstract searches with words, phrises and stenis work well If one has 
^^_4xreat1ve ^ind, (3) vVSS should have had MeSH tree'- structures fof BT/Nif terms 
^T(n^te: they Were not supplied to l^tteVle), (4) scbpe notes were not working, 
• and'{j5) VSSjJs ohly as godd 0 ifm vocabularies that went Into It.. 



10.0 qfeng USERS SUWEY aWMffE 

In 1979, as" part of " « previous -NSF grant on VSS. (10)/ a national 
surveiy lias conc^:ted to- detenilW SMfchlng patterns and preferences of on- 
line users and to* eltcit tteir .reaction to 'VSS. There jnere several key 
^uesticms In the survey refated to thfs^ research^ over ^nd above those 
questions pertaining to VSS. One^. dealt with the frMuency .df searching *f or 
infonaation using witiple data bases. ' Another pn^ for * the reason Mhy 
searches were condu^ted^ against, single date bases, finally, s^eral questions 
addressed the issue of seardiing With controlled, uncontrolled, and a 
conbination. of controlled .and* uncontrolled' tems. . 

It was felt that by rerunninr the %8m iurv^y at this t\m (four 
years after the first one) a unique qpportufrlty was aveilable to observe any 
shifts .In- user search patterns and/or preferrence^ with t1«e. Ver> few, if 
any, surveys in this field are r^ated except for salary' surveys. A.benchmrk 
question for the 1979 and 1983 surveys was the nonthly <k)11ar expenditures by 
' 1ndivfdual# and organizations for' on-line searches. It^'has been widely 
r^orted ahd generally accepted that the w-line business is growing at the 
rate of about 30 percent coiflpounded annually. If our date confimed this 
- observation, then any other observable shifts in on-line search patterns would 
have added credibility. . \ « 

10,1 WetftodPlogy 

. ^ _ / ' . — : ' ^ • . 

The surv^ wJMr^ated by reissuing the 1979 questlfNinaire (un- 
modified). To lalnlnlze ^tributlon costs, the survey was distributed as the 
1983 National On-line Meeting; held in Nm York. (See Appendix H for siaple 
questionnaire.) Meeting organizers granted permission for a passive 
distribution of the survey^ near the registration desk. Mseting registrants 
helped thmelves to a blank survey quest ionnaire« at the tiott of 
registration or any other tine during the aeeting. About 325 questionnaires 
were distributed, in this oanner; 38 valid questionnaires' were retuVned, for a 
12 percent rate of return. Consequently, saiaple sizes for 1979 and 1983 
surveys varied considerably, 755 versus 38, respectively. 
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All partlijipation was voluntary ^and responses mre kept corvfldentlal. Data 
iiere analyzed via staple descriptive statistics. Finally, the 1979 s*irv$y data 
»<ere reanalyzed .subsequent to our 1980 report ^(10) because 66 additional 
returns mrt received after^the publication cutoff date. This report reflects 
all the 1979 data, or 755 valid returns. * 

V 

10.2 Results . 

• . » . ' r 

Benchmarks for conparing the two' surveys ^re- the questions about 
'individual and corporate expenditures for pn^line s^aihch activity. The data* 
for .on-11ntf search expenditures (by individuals) confirm . various open 
literature sources that esti^te that' on-line. searching 1s growing a>the rate 
of 30. percent coapounded annualty (Table 3(k:)* However, the data 'als(> suggest- 
that corporate search activity is growing at an even grea,t#r rate, 45 percent 
confounded annually (TabU 31c). Of these two benchmarks. Individual sesfrch 
activity Is probably the more reliable' because- individuals' generally are In a 
better position to estimate their own search activity than that of their 
overall ~ organization. Some respondents adaiitted theiif* ancertainty; about 
aggregrate corporate search activity; -others simply omitted a response to the 
cor|»rate questions. In any case, ttere Is reason to believe that 1983 survey 
results, albeit a small sample, have some validity since they close ly ag^gpe 
with reported literature regarding growth. 

Conn6ct-hour growth for Individuals Increased at a lower fate, 18 
percent com^nded annually (Table 30b) and the nuirfier of searches per month 
increased only slightly over* the , four year period (Table 30a). It can be 
observed that on-line searches averaged 0.25 and 0.44 hrs/searjh for 1979 and 
1983, respectively (Tables 30 and^ 31). This observation suggest! that one of 
the. factors contributing to on-Tfhe Growth is lengthier search sessions. Other 
factors, of course, are, price 1ncreases*and the nwaber of new custo^rs. 

Figures 5 and 6 show the fmf yB&f tref^s in search activity. There 
Is* a noticeable shift, left to righf, ' In .nuiAer of searches, connect hours; 
and dollars expended per month In these two figures, fconf.lrming- th6 growth 
dynamics reported in the literature about this business. 



74 
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Another as|fect abCNJt the online business Is search activity within 
various sectors of the user cowaunlty. Table 32 shows ,1979 search activity by 
four major user sectors. This breakdown was not previously r^rted In our 
1980 report but was produced during our re-analysIs of all 1979 data. However. 
1983 data were not analyzed In ^1s fashion 4ue to the stttll Initial saiaple 

size. * ^ . 

It can be seen that ^'vemiient sectpr users did nore searching ip 
1979 both at Individual and organizational leJiaJs than ,any .other sector^ 
However, the' for-profit sector user spent more mpney for .^searching, on the 
average, than any of the other sectors. 

. ' ^ the Individual level, both goverrMeitt and university 'sectors 
spent about the sam aiawint on an "average" search ( $7,75), whl^e for- 
profit and iKMi-prof lt sectors spwit about $17.89 and $10.82, respectively, 'for 
the average jsearch. At the organizational level, university users spent about 
$8,fe per search,* government and non-profit users spent about $11.40, and for- 
profit users spent about $27.52 per swrch. ' ' ' , 

Wien asked who t*as paying tor on-line searches^ (Question 6 In the 
survey), a blwodal distribution resulted (Table 33). fhls was fexpected, 
b^ause if a respondent relied that end user$ were paying for searches 100 
percent of the time,, t^n, of cwirse libraries pa.14 0 percent of the t1«e. 

, If blanks are counted as 0 percent, i^ich was virtually always the 
case on the Individual questionnaires, then Table 33 shows that eiid ustirs are 
paying fgr a greater percentage of the searches in 1983 than they were in 1979 
and, <jconversely# libraries are' paying for far fewer searches in 1983 than In 
JL979. ' ' . : • , , . 

Restrictions which a*n end user aiight-place on an on-line search, in 
descending order .were: (1) Jiait the search by date, nunbfer of cita<t«>ns» etc; 
(2) limit the search to soift certain data base(s); (3|'l1fflit the search by 
^om cost celling; and (4)' Halt the *search by some amount of » connect time 
*■( Table 34). Data foif 1983 tend to reverse the order of (2) and (3). 

On the question of which on-line retrieval services pe<^le are using 
(Question 8),' there has been no change In the raiAing of major services omr 
time (Table. 35). 
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Individually, Dialog apparently was being* used sonie«tfiat nore 

* frequently In 1983 than in 1979, lihile, fbr £lhill (HLM) and all others there 

• was either no change or somewhat ^fess usage In 1983 compared to 1979. What Is 
iwt reflected is the usage of new. servlc^^^^ wany* ^^so^ 
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the respondent populations varied heti«een the 1979 and 1983 surveys Mhlch 
would tend to distort the results. 

One of the primary questions of Interest in this survey was the 
trend In single versus- multiple data base searching patterns. A shift toward 
greater usage of oultiple data bases . represents a Justifiable case for a 
search aid M\^e VSS. 

Figure 7 shoirs how multiple data base search patterns are shifting 
Mith tine. Single data base searches are on the decline — note the shift from 
right to left In Figure '7A. Searches involving two data bases show ininor 
shifts In individual categories, but overall, appear to be unchanged (figure 
78): 3-data-base searches appear to be increasing with time (Figure JC); and 
inultlple data base searches^ Involving four or sore files are definitely 
Increasing (Figure 7d). In fact,* respondents who answered this question for 
the 1983 survey fr^ently change the choice on the questionnaire by adding 
the word "wore" to the choice identifying four data bases. Overall, the' trend 
is toward tore aiiltipl^ data base searches. ^ 

VRien asked uilder what ccmditions searches mre limited to one data 
,base (Question 10), the three most common reasons given in decreasing order 
were: 

(1) Exhaustive search was not required o v 

(2) Nujtl-base searches are too costly 

(3) Multi-base searches are too time confining \ 

* ' * • . \ 

These results Ignore "other" as a reason (^Table 36). . , • ' ' 

, , The fact that some data bases might be tod difficult or too 
unfamiliar (Column d. Table 36) ranked fourth and fifth in 1979 and 1988, 
respectively. In other words, users are not confining a search to one data 
bas^ Just because others, liilght be unfamiliar or different to use. Overall, 
there was virtually no change in user response to (^estion 10 over time. 

On the questim of which subject areas w(»i1d be most useful for 
multiple data base searching (Questiw* 11), it Is apprc^jriate to examine the 
results (Table 37) b0th in terms of the number of respondents per subject area 
and the mean value of the responses because the usefulness of an area Is a 
function of both factors. . 
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In 1979, respondofits rartced the life sclence^/aedlclne area highest 
among subject areas i*e1re nultlple data base searching* Is (or would be) most 
useful. This area<.a1so received, the greatest nuwber of responses, 259. Other 
areas receiving over 2{K) responses are shown In Table 37 1n decreasing order 
based on the mean. 

In areas, where the response iras between 100 and 200, education 
received , the highest ranking, whiles In areas receiving less than 100 
responses, agrlciilture was rarriced the highest. Mathe«at1cs was the least 
popular area (N-13) and the area least' likely to be useful for miltlple data 
base searches (fliean"2.38). 

The 1983 suryey was too aiial4 to provide a useful cc^parlson with 
1979 data for Question 11; none-the-less, t\w distribution of 1983 data Is 
given In Table 37.' It can be seen that areas receiving heavy response In 1979, 
were also receiving the greater percentage of responses In '83. 

Data for Questloii 12 were -not coiiplled for. the 1983 survey because 
there were too few resffffndents and too aany possible data bases and 
coi*1nat1ons of data bas^s to warrant the effort. The 1979 data we^e reported 
earlier (10) for this question. 

(Kiestlons 13 and 14 dealt with the three principal raethods for 
tonduct,1ng subject searches: controlled vocabulary, free text and a 
cflBiblnatlon of both. On these two questions, we bad the benefit o^ additional 
data for lOT because the 65 participants In the field evaluation of V?S were 
also asked these sasie questions. 

Figure 8 (A) shows a definite- shift frow right to ]eft over time 
indicating the decreasing popularity of controlled vocabulary searching. 
Figure 8 (C) shows a very slight shift, right to left. Indicating virtually no. 
change in free text searching over tiae. 

On the other hand. Figure 8 (B) shows a gradual shifty from left to 
right Indicating the increasing popularity of searching data bases with a 
coablnat ion of controlled vodabulary and free' text terms. * . 

Table 38 clearly shows that the users prefer a combination of'' 
controlled plus free text searching, follrfwed by free text searching 
(exclusively) and :controlled vocabulary'' searching (exclusively),. ^Actually, 
tfle latter two methods exchanged places over time. 
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TABLE 37 'SDB.KCT /IKEAS HOST U^FUL FOR 
WinPLE DATA BASE SEARCNIII6 . 
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TABLE 38 USER PREFERQKES FOR TYK 
OF SBJECT SEARCHIie DOiE 



Us«r Preferences* 
Seurdi Itethcid 1979 1983 

« 



i. Controllfiid plus 1.31** 1.26 * 
free text (658) (101)*** 

2; Free text <mly 2.38 2.23 

(634) (100) 

3. Controlled wily ' 2.28 2.49 
* ' (632) (98)- 



*Scale i«as 1 to 3 i^re: l"^st 
preferred; 3«least preferred 
]^ar1tl»et1c nean 
***nwpber of responses 



When asked In what percentage of their present searcbfes ^ user wight 
turn to a subject swltehlng systea (C^iestlon 15), given that the only exposure 
a respondent had to such a systen was .a <;wer1ng letter containing a s1in)le 
examle. the response was evenly div.lded In 1979, but Is skewed toward the low 
end if the usage scale In 1^ (Table 39). The only clue as to why this 
response shifted over time way be due to the overall decline In user 
preference for controlled vocabulary searches (Table 38). 

If there Is a.dlrclft relati<msh1p between th^ low user preference 
for control led-ters searches and a low, aiftlclpated use of VSS, this suggests a 
possible misconception about, how VSS wight be use<^. The misconception 1? this: 
since VSS^Is essential ly a data ^sb of^ control led . vpcajMlarles, It Is only 
useful for controlled-term.searches. Thlls. of course. Is 4iot true. VSS -output 
•can, and probably should, be u^ to conft|ruct searches #»1th c^trolled terms* 
free text /ten^, ariii controlled plus f;fe^ t6xt 'terras.' Eoar 0xai^le, If a'user 
1s'lntera6te/ln Information related" to; the term'mempry, llii the psychological 
sense, M VSS produces "human Information >#a§e*. •fpfge^tlng", -cognitive 
proces^ies", detention", etcT., all of the^ controlled vocabulary terms can be 
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used as Is for a controlled-teriB searcb, -or they, can be broken down Into 
Individual words and used as free text terras. 

If "retention", "forgetting", etc, are valid terms and the .user 
Insists on searching In a free • text aode, then VSS has served the same purpose 
as an abstract, title, or descriptor field, namely as a source of additional 
search .^erms. It Is evident by some responses to VSS tJvat some people- cartnot 
or will not make the transition that controlled vocabularies do not 
neces sari ly .have to be used for controlled descriptor searches. 



TABLE 39 ARTICIPATED 
USAGE OF VSS 



X VSS 

Usage 7d 83 



0-120 


164** 


. 13 


21-40 


• 138 


7 


41-60 


•162 , 
123 ' 
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61-80 
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81-100 


106 


3 


> 40 


56% 

♦ 


44% 



**ffciiflber of responses 



Unless a user exhibits a high degree of 1ng«iu1ty. no Inverted list 
will produce "retention" or "forgetting" for the word "mQ»ry". The user In 
this case Is left to his own devices or required to read abstracts, titles and 
descriptor fields In retrieved docimienr. sets to find additional clues. The 
point Is, much "intellectual, effort . at^relatlng concepts to each other hsfs 
already been expended developing controlled vocabularies. Subject switching 
NmeHely taps the potential, whigh' is present In feach c<»itrolled vocabulary. 

Question 16 probed for the possH|je migration from single iJata base 
searches to multi-base searches assimilng users had a search tool like VSS. 
Table 40 'Shows th/it about 1/3 of all respondents would consider converting 
their single-base searches to. nu It 1 -base, searches at the rate of Aout 40^ 
■per:cent of all their searches, tills response was unchanged with. tine. 
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The dfesponse to Quesflon 17, '♦Factors affecting the use of VSS*^, is 
shown Mn Figure 9. "Vocabulary dIfferencesV Is the reason most cited Jor ^ 
user's Anticipated usage of VSS, both In '79 and '83. Cost of uslnj^subject 
switching ranked second' followed by the number of data bases being used in a 

search. . • ^ 

r ^ the question of what' users would be wilTIng to pny for a subject 

switching capability ((^je^tion 18), their response Is unquestionably very 
little, coii|)ared to current connect-hour charged for comserciany available 
data bases thoaselvesJIsbTe 41). It appears fiiat this questlop has'^cftanged 
little ovef tiine^ * * 
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Finally, on the question of just when a user would turn to subject 
switching for assistance based on the. niuaber'of data base»-beinsr searched, the 
trend is 'toward four or norg)data bases (Figure 10). • 



TMlE 41 l^*S VllLXlCiESS TO PAY 

FOR suKscT suncHiie 
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This fcoincldes with^ the trend toward a greater number of Multiple dat? base 
searches Involving four or- sore data bases (Figure 70). Thus, wuUiple (|ata 
base searching trendy are increasing in the right direction to justify using a 
subject switcTiing capabilltif yet there is evidence to suggest soaie resistance 
about using such a systen (Table 39). Some of this resistance can be 
attributed to the fact that a certain percentage of searches ddh't require an 
axhaustive,"^ BBilti-base search (Table 36a),. Another factor for possible 
diminished use of VSS is avoidance of additional time and costs associated 
with multi-base searching.' (Table 36b & c) which is prin»rily a resistance 
unrelated to the VSS systoa itself. »iat* users are saying is that a nwlti-base 
search involves more of their time, more on-line costs (for each data base to 
be searched), more off-line print costs, and perhaps irfere post-search activity 
such as sorting and organizing of retrieved references in order to eliminate 
duplicates. ' . , 
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11.0 COKEPTS FOR EXPRESSIOW-LEVEL SyiTCTIiiS 

The car rent VSS systea switches on vocabulary terms one tenn.,-at a 
tiffle. That is, if the query contains two or wore teirms and Includes Boolean 
operators, each tem «ist be entered separately, Into VSS. Following switching, 
the usei^ najst construct his/her own search strategy using the VSS output anjj. 
appropriate Boolean logic , « 

" To streamline the process, it would be iteslrable.tg perform subject 
switching on the entire search strategy (expression).* Such a system was 
conceptualized but not Implemented. 

In expression-level sw1tch4fig, VSS would have the capability to 
"understand" the Boolean expression supplied by a user and would construct the 
appropriate search logic for each vocabulary' specified. As a slB^le example, 
assume thens are vocabularies; A. B, and C. A search, for CAR Indicates the. 
following: • , ' ! - * 

1 * AUTO (B) • ' ' - 

AUTOMOBILE (A, C) / ' , 

• CAR (C) 



m 



This inplles that AUTOMOBILE Is the controlled term for vocabulary A, AUTO 1s^ 
the controlled terra for vocabulary B, and both AUTOMOBILE and CAR are used for 
vocabulary C. Similarly, assume a search of LARSE yl^elds. 

. . Bia (Aj * , - , V 

LARGE (C) . - - ^ 

This Implies that there 15 no corresponding concept- Indexed in v^abuiaT-jr B. 
In expression le^el switching the user might enter:* - 




BtG .and. CAR wher« .and. is.a Boolean "AND" 



VS^ wu^^ respond: \ 



j|^|^foMOB: 
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LARGE .and. AirmmiLMC) 
LARSf .and, CAR (C) 

* ♦ 

Notfe that since the conc^t of BISness did not appear lln vocabulary B». no 
search expression using AW) logic c'ooW be forced. The user could then revert 

.to' a sli^gle .term search or attempt the search on an uncontrb^led field 

.(titles, abstracts). . . 
, ^ Similarly, if* the user had entered: 

BIG .or. CAR ighere .'or. Is a Boolean "OR" 



VSS would respond: 



it 



BIfi .or. AUT(»mLE (A) 
AUTO (B) 

^LARfE .or. AUTW«»ILE ,or. CAR (C) 



In this case Tack of a natch Is not a problem since a Booleai|. "(»" Is 
Satisfied If either term can be found. 

In a similar manner, more complex boolean searches could be 
switched, l^sw to best handle a. missing AM) concept in a complex query Is still 
an op^n research question. " . 
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, 12-0 smm im cowausiois 

VSS Is the Biost advanced tool yet developed fon searching on-line 
bibliographic data bases based on controlled technical vocabularies. The 14 
vocabularies in VSS represent an investment of about 52 man-years of creative 
work by- the original vocabulary developers. This stored knowledge base was * 
evaluated as an aid for structuring and enhancing search strategies. 

- 12-1 Controned VocalMilarles ^ 

The use of controlled vocabularies in today's search environment 

cannot be denied. The survey data shows that the nia^er of searches involving 

. btoth controlled and ^ free text, terms is actually on the increase, i^ile 

searches involving one or the other approach, exclusively, are rather 

unpopular. This finding is reinforced -by the fact that users actual ly»prefer, 

by a wide itargin, the co«binatibn of controlled plus free text searches over 

ejlther approach used indlHdually. '^'^ '\ . ' 
^ Tlfc fact that searchers not(onir***e control led/^lus\frfee-text te^ 

in an increas?hg 'huafcer of their selrclTei but actually prefer , this approach 

Uells us that this pheno^n^ 'is f»t simply a case of "blind faith" usage of 

controlled Sescriptdrs Just because they are available. If that were the case, 

the usage response would differ from the preference response. 

On the basis of these findings, wfe conclude that users perceive no 

superior indexing method tn IS&R systems, opting instead for the synergy of, 

two methods combined. 

12.2 HUltlpTe Pata-Basc Searching 

Multipl«^Ha-base searching patterns are changing with time. Th^ 
/trlbnd is toward increased usage of multiple dat*- bases . in on-line searches. \ 
This trend suggests the need .for search aids which transcend data bases or 
mitigate the differences among them. ^ n , ' 

In other w>rds, users need navigational aids to search mri 
effect1v*ely across diffiirent data bases. Better «^an 75 percent (42 out of 55) 
of those »rho participated In the VSS field evaluation responded favorably, to f 
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the conc^t of subject switching, suggesting the wide appeal of a navigational 
aid. 

% Probing deeper Into the nultlple data base searching iss'ue, but from 

the v|ei^o1nt of user Inhibition due (a) to the plethora of' available data 
bases or (b) to the possible difficulty of using less familiar files, our 
survey showed that neither issue was a serious factor for confining a search 
to a sinple data base. Instead, users fnfl^cated that when searches were 
limited to a single data base It was^ primarily because the user did not 
require an exhaustive search or -there was a cost limitation which precluded a 
broader search across multiple files. 

It Is concluded that the popularity and user preference for 
controlled plus free text searching and the favorable response to subject 
switching as a *iavigat1onal aid for cross-file searching lends credibility to 
the approach being pursued In this research. 

12-3 The VSS Sfstem 

Feedback from the field evaluation provided valuable, insights into 
the strengths and weaknesses of VSS as a ^stem'and conc^t. Users who thought 
VSS would make their job easier (about 40. percent) felt that It would save 
them time and money because they would have ffiiwr printed thesauri to consult. 
They also liked the juxtaposition of terms from various thesauri and thought 
VSS would improve their presearch preparation. 

About half (45 percent) of all participants expressed an overall 
positive attitude toward VSS or thought it was interesting but needi more 
work. Many (about 75 percent) participants responded positively to the concept 
of subject switching. • 

However, several weakness were also observed. The menu^ approach to 
-*VSS, with no prov1sion\for direct access, led to user frustration. This was 
exacerbated by the unforeseen high usage of 300 baud terminals 1n the on-line 
coraRwnlty. Menu-driven syiteis are npt amenable to, slower terminals. Also, 
once users .become' familiar with a system, m^nus are not necessary and thus 
unacceptable. , - 
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Users felt, and the Investigators concur, that vocabularies must be 
kept current withjrapidly changing technology, in order to meet the needs of 
searchers antf end usets. This Is a potential problem for vocabulary developers 
and a, real problem for a system like VSS. The ySS vocabujarles used in the 
field evaluation were 3' to 5 years old, due s1n|)ly Ao the cost of reformatlng, . 
reprocessing and rebuilding new files with tS new versions of the 

vocabularies. * 

Users expressed a need for more * synonym and hierarchical 
relationships than were ^ovided. Users also wanted scope and history notes. • 
The former is a shortcoming of vocabulary suppliers and is not easily 
rectlfiable, while the later 1^ a , shortcoming of VSS itself and Is easily 
rectlfiable. 

Another problem, was the |K)or quality of output derived from sternnlng 
algorithms and< Inverted-file adjacency features. These types of switching 
options created more noise In YSS output than useful terms and were 
undoubtedly responsible for establishing a negative Impression of the system. 

in the minds of sbm^users. * 

Based on the feedback provided by the users, we believe that with 
additfonal wort, a system such as this can achieve the potential usefulness 
that the usiers and the Investigators expected. ' ' i 

12.4 .Perforaawcte of IfSS Hocfailes 

^ ^ • . 

On»one hand. It was shown that the mflst SHCcessful switching, based 
on a' performance measure which takes into consideration the amount of output 
and its relevShce, Is a function of the-* similarity of the vocabularies (see 
Section 6iO). VocabulaHes^that are^^llar in^ syndetic structure and subject 
content produce more satisfactory switching results than those that are 
tHsslmllar in one respect or another. 

In this version of VSS, a tejpm-level ^nalysis suggested that the 
social science module would give consistently better perfo^nce *than the ■ 
other vocabulary modules, followed by the business module^ the .physical 
science aqjlule, and finally, the life science module. 

There was a direct relationship between module performance and 
subject similarity among the vocabularies. One measure of subject similarity. 
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Is the 'rate of exact matches found during switching. Ulso,' It, was observedSr 
that t|je social science vocabularies were very similar to one another with * 
respeet to their syndetic structures. 

On the other hand, module perforaiance rankings were quite different 
In the field evaluation by ' Intermediaries (seC 'Section . §.0). Her^, 
intermediaries were asked to rate VSS on a scale of 1 to 5 fot* each search 
term entered (1 meant V$S was of no help, 5 meant VSS was very helpful). In a 
mixed model analysis of covarJance the module rankings were as follows: 
physical sciences (2.4J), social sciences (2.38), life sciences (2.12) and 
business (2.05)'. > - 

The principal reason* for ^Jie big difference In module ranking© . 
between the formal analysis and the field evaluation was the inclusion bf 
related terms in th^ fi^ld-tested version of VSS. These terras were not present 
in the version used to conduct the- formal analysis. The top t^(0 modules rated 
by intermediaries, physical a^ social* science, contained vocabularies that 
were rlcfTlt^related terms. * ' . 2^ 

The performance difference described above shows jii^^ow 1iH)ortant 
the related term is, and why VSS is now best viewed as a tool tor provide a 
shopping list approach to searching. 

AHhough the formal analysis of switching strategies served a useful 
purpose in planning pre-defined strategies for subsequent use in field 
experiments, user ratings represent 'a" better indicator of VSS system 
performance. It also appears that for maxinwra usefulness, ,all of the original 
syndetijc structure should be incorporated Into a system like VSS, including 
relate*! terras, scope and history notes, subject category codes (e.g., COSATI^ 
codes)! and any other special relationships provided by the^ vocabulary 
supplier. ^ . * ' * ■' ^ 



* , 12.5 End User Exper1«»its 

VSS was evaluated at three sites in actual retrieval situations to 
estimate its effect and Impact on end user satisfaction. • Experiments were 
designed i^ereby end users evaluated citations retrieved from a normal search 
with those produced from a VSS enhancement or modification to the same search. 
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% .-Two types of experlaents were designed,; Iterative searching and 
parallel searching. In the Iterative searching* experliient, a normal search was' 
repeated subsequent- to jts nodlflcatlon by VSS. In the parallel experinient. 
two Intenaedlarles worked on the sane end user request beginning with the pre- 
search Interview. One Ihteraiedlary conducted \ norwal search^ the other used 
VSS. 

iS% perfomed quite well In the Iterative searches. In both 
searches,, end user satisfaction was high and VSS produced a significant amount 
of relevant output. Inf" one search, VSS accounted for 20 percent of, all' 
relevant output, in the other search, 62 pefcent; * . ; 

, ' In ^he experisents * where a parallel approach was employed, VSS 
performance was about as go<Kl as non-VSS perforaance but end user evaluations 
appeared to be /Inconsistent' and lllogfcal. Of the six searches conducted via 
the parallel design', us^rs gave VSS searches a higher rating In two, Non-VSS 
searches a higher rating in two. In two searches the ratings were equal, the 
success of these searches appeared to be dependent on Interpretation of the 
end user question rather than the use of VSS. Therefore, it was conclud«i that 
perlments involving two intenaediaries probably confounded the issue. 

/Vs. it, turned out, *the parallel' design was a difficult undertaking 
becaise. search intentediaries ^eoed hard pressed^ time in their real-life 
wrk enVirorwents. Many foui^ it difficult to start ,the|fexperl8ient and nobody 
completed all the searches planned' (six per site). Que site failed to 
participate, at all even though liiey *«re briefed on the exi^riment and trained, 
on the system. - 

/ It appears that' 'tiie iterative design is the most appropriate method 
for evalbatlng the effect of VSS because only one intermediary is involved in 
the search and by n»dify1ng his/her original search with VSS terms, a clear 
distinction bfetween searches is possible. 

Several additional observations can be made about these end user 
experiments; ' 

(1) High-recall type search strategies can lead to some 
rather low search precisions at times, (e.g., 2 and 3 
pefxent). However, user toleration of noise at these 
levels may be quite high. 

* m 
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(2) High-recall type search strategies dSS't always * 
produce iilgh recall. In several Instances, VSS . 

\ produced a significant amount of additional uhlque 
-relevant citations even though the Hon-VSS search was 
. - ''geared for hlflt recall^ * , - ^ - 

* " ' ' ' ' 

(3) Search strategies can be a very coaplex series of 
nested stateaaits. It appears that sone of these, 
nestings, and Boolean coeblnatlons are slisply done to 
reduce fthe documt set to a aanageable mMrt>er. 
Therefore, what, IS thought to be a valuable data base 
attr.1bu^,-ne«ely Its retrospective depth* tiiyns out 
to b^"^ liability In certain high-recall searches, 
where aental gymnastics , are required just to Identify 

a set of citations i*1ch the user can (a) afford to - 
print out and/or (b)^cfpe 'with (Mice It Is delivered. 

" . ■ . 

12.6 Evaluation ^ fS& bv IwtafiMiiarles ^ - 

ft. ■ * * 

YSS was. evaluated by 65 Inforaatlon professionals, from brokers to 
professors, Indexers to searchers. The evaluation consisted of a packet of 
materials explalrt^ftg VSS and how to access/ It on-line, a user evaluation form, 
and a scheduled week In idilch ta log-on' and use the ^te«. Private, 
govertni^t, and acatalc sectors wer6 represented. Overall, user proficiencies 
on the VSS data bases raAged from Just average to virtually little or none. 

Seneraliy. the participants thoii^t VSS was easy to learn, use and 
understand, bMt they were less certain about Its capabilities and output. 
However, 40 perc^it thought VSS would make their job easier irtille only 9 
percent felt It would make their job harder* Also, participants with fewer 
years of on-line experience and those with higheif' degr^s thc»jght VSS would 
make their jobs easier. This finding was .based on a Pdarson correlation 
between experience dr education arki performance ratings given. • 

The participants tried 623 searches In VSS. Iteable* results were 
obtained In 62 percent of the att^ts, and 9f these, about 22 percent 
produced six or more Useable ;ter8» per entered term. Although the overall 
l^lng for VSS was 2.25 on a 5-po1nt scaJe this ratli^ Increased to 3.08 lAen 
dnly. those searches whtch produced usable output were considered. Also, the 
average rating Increased directly with the amount,of usable output. In searches 
i^ere 6-10 usable terms w«fre produced the ratings averaged 3.6, and where more 
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tftan-lO usable terms' itere^ produced, the racing average^ -4.1 on a 5-po^nt 

scaleJ • • ; " ' • 

It Is obvious that a system such as VSS Is only- as good as the 

vocabularies In It. It's performance Is directly' related to .the fiunber, 

•specificity and currency* of such vocabularies. The ^larger the system, and the 

We up-tp-KiAe the vocabularies, the better the performance. It ajpears. that 

high performance ratings afid high user acceptance are ¥»ell within the grasp of 

such a systoi. ' . ' 

A mixed model analysis of covarlance was also employed to express 
the VSS" rating assigned b> a participant (to each of the 62^ searches) as. a 
function of (a) VSS module chosen, (b) «ie participant's proficiency with a 
particular VSS module, and (c) the unique qualities of e«ch Individual^ 
participant. The model asswaed that the average ratings asslgifcd to searches 
In one VSS module may differ from those assigned In. another, and It also 
as&tmed that VSS ' ratings change linearly with Increases In"^ participant 
proficiency within a particular module. The 'unique <n«11t^es of individual 
participants were assi«ed to be ranctom. • • 

ResulU >flid1cated that rflT factors were statistical ly^signlf leant 
but their est1mJ(&» eff^ts differed. In decreasing order, the relative 
Importance of the factors were; (1) uncwitrolled factors (from search-to- 
search); (2) unique qualities of the participants; (3) ttie participant's 
profltienty with a VSS module; and (4) the VSS module chosen. 

The effect cfeie to a participant's proficiency was found to be 
Inversely related with his VSS rating. In. other weirds, pafticlpants rated 
themselves proficient in a given module rated VSS lower on^the average than 
those who claimed to have proficiency. 

The effect due to module Or subject area was the least important of 
the variables studied. A spread of 0.4 points was observed between the highest 

ft 

and lowest rated mrKlules. 

Finally, there was a considerable amount .of variability in VSS 
ratings that cannot be explained. This variability represents the uncwitrolled 
factors in the experiment. The size of this variation was estimated at +2.00 
points. ' . ' ' 

In o|^-ended questions, about four and one-half times as many 
participants thought VSS would make their Job easier as thought VSS would make 



their Job harder. . Cipse to half of all participants thought VSS was very 
valuable, valuable, \pi: "Interesting but needs «ore iwrk". The concept of 
subject switching a$ a search' tool was very well received; about three-fourths 
of a.l\j)art1c1pant$ responded positively to the approach. 

in view of these results, it was concluded that subject 'switching 
has a^t^tfally wide ^fH^eal to Informatloii^rofesslonals, byt VSS irlll need* 
several luprovements or even wajor :rede%jgn If It Is to address their needs - 
and concerns. These needs,' briefly ' swwarlzed. are: (1) Include sore, 
vocabularies; (2) keep the system updated -with current . Versions of 
vocabuTarles; (3) Include Vt^ full syridetlc structure of each vocabul^; (4) 
provide direct access as an alternative to aenus; t5> eliminate stenmlng and 
adjacency features; (6) allow users'to coablne vocabularies In any combination 
desired^ (7) nalce It Inexpensive to use. 

12,7 Oft-fclne Users Survey 

• r I. # 

^» 

^ The user survey provided Insights* Into how seircMng patterns, are 
shifting with tiale. Areas deaTing with controlled vocabularies and multiple 
data base searcfiing have already been covered earlier (Sections- 12.1 . and 
12.2). . , . 

^ On-line expenditures by individual searchers have Indeed grown at 

the rate of about 30 percent c^ounded annually. This agrees with various 
market study results reported in the literature. The average monthly on-time 
expenditure per Individual searcher was $972 In 1983. compared to $377 In 
1979. However, spending patterns differed, fnom one ei|)loyment sector to 
another. In 1979, the avera^ monthly on-lflie-expencMtures by Indiv.iduals by 
•sector were: for-profit sector ($467), government sector ($398), non-profit 
sector ($276), and academic sector ($222). There were no surprises here except 
to show Just how low expenditures Ibr on-line searches in the acadesic sector 

f 

• are relative to the other sectprs. The 1983 survey was not large enough to 
provide reliable figures by sector. 

A greater percentage of end users is paying for his searches today 
than in 1979. Conversely, library budgets are supporting fewer end-user 
searches today ctm^pared to 1979. 

\ 
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In 1979, respondents indicated that the coinnon type of 

limitation (-if any at all) placfed on a search was byli&e or number of 
citations. The next most canmon liroitition was by dat# baite, followed .by a 
specified cost ceiling. -In 1983, the- most C8l6)n 11m1t«.*pn placed on a seircb 
was data -base, followed by tiate or number of citat'jons, and finally co^t 
ceilihg. ^ . . • . 

The usage pattern for seven major retrieval services showed very 
little change over the past four years. Dialog was st^ll the most fr^equently 
used system, otlC the Jeast frequently us^. Also, there was^no shm in the 
relative standings of these majoc systems based on the usage question. 
However, nifflierbus new systems: were identiifiid, the most frequently mentioned 
being NEXIS, CAS 01tt.INE, and DOM ^S. ' . ' ' 

•There has teen. a shift over the past four years in the subject areas 
where users feel multiple data base search 1 woii Id be most useful. In 1979, 
the three top areas were:^life sciences/medicine, agriculture, and engineer- 
ing, respectively. In 1983, r^bndents rated j)sychol(5[y, , business/economics, 
and life^sciences/me'dicine the^ top three choices. >| 

With the simultanec^s decline in (a) searches using controlled terms 
exclusively and (b) the respondent^ anticipated use of VSS over t4»e, a 
possible cause and effect relationship is seen bet««en these two questions 
which leads to the conclusion that there, is' probably a misccmceptlon about how 
VSS raiglit be used. 'The misconc^tion is that VSS, being based on controlled 
vocabularies, must be useful only for controlled-term searches. 

On the contrary, VSS is useful for all types of subject searches,, 
controlled or free text because VSS is approaching the breadth and depth of 
unique words and phrase's that title and abstract fields contain. Fields that 
are rich in technical terms are the ones that searchers turn to when "fine 
tuning" a search. We believe that VSS is a rich source of technical terms to 
be used" in any one of many imaginative ways during subj^ftjearching, 
regardless of the approach being taken. In other words controlled terms can be 
used in free text searches and vice versa. 

(ki the migration question, about 1/3 oj the users indicated they 
would expand a single data base seaf*ch into a Multiple dat^i^base search 
greater than 40 percent of the time If they hjd a systea like VSS. However, 
they would not pay much more than about $10 per connect hour for a system like 

* 
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VSS. Their need >or VSS Increased as their need for more data bases per 
search Increased. The survey showed thtft the trend In swltlple data base 
searching Is in the direction of 4 or aoreNiati bases per search. 

Users see subject switching as a valid, useful concept, but one that 
>ttie^ should not have to pay wich for. As'sotae of them see It, the on-line 
vocabulary system has to coin>et^ with *cheap, off-line, printed versions of the 
same thing.' - 

th6re IS no^q^stlon that a systew like VSS can be designed, with an 
efficient and streaallned user Interface, 'larger and wore up-to-date 
Vocabularies, and ^ven mre of theai.' Itowever, the question of greatly reduced 
online rates Is a mrketlng and busltiess decision Involving the data base (and 
vocabulary) suppliers and the providers of on-11fie retriJtal services. % 

It Is believed that in tise, users would become very efficient using 
a systeffl like VSS, so the extra cost incurred by using VSS ultimately may be 
very small f'elative to the total cost of the search. 

The benefits a^e reduced search-preparation time, iirproved search 
strategies and retrieval, and greater usage of existing data bases. Therefore, 
all parties In the on-line ie&rch scenario derive some benefit from a 
navigational ald^such as VS^ If the benefits are substantial, on-linl vendors 
and data base producers could afford to reduce or give away the navigational 
aid on the theory that more revenue will be generated via greater data base 

• * 

access. r * 
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103 ^ . 
13.0 RECOlAEIiMnOIG 



On the basts of* this research and the feedback, from users 
evaluated VSS^^ the foil owing Is re$on»nded: 

(1) Build an entirely new model of. VSS, based on the 

relatiwal data base model as a solution to the 

update problem and therefore, .as a means of 
maintaining current vocabularies In the system. 

• - ♦ 

\(2) Expand the breadth and d^th of VSS by Including many 
more vocabularies and- all of the syndetic 
• relat^shlps available In them. 

• (3) Streamline the user Interface" to permit, rapid direct 
access to VSS files and Eliminate non-productive 
features (e.g. stMlng and Inverted file adjacen- 
cy^). , 

(4) Consider storing VSS on ^ videodisc In -digital * form 
and perform all navigational tasks.^ 
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•* . * ■ ' . - • 

At th| end of Grant IST-7704498 six- .full scale vocabularies (DOE, 
EI, INSPEC.* NASA* and- tw frSm Chemical tostracts) were included In-' VSS. 
During ^rant-IST-79-U190 several ntk vocabularies were' acquired and converted 
froiB the vendor's fbnnat to a standard format used as Inpu to the VSS filer 
".fcjjulldlng programs. The new vocabularies were: 

- I 
ABI INFORM Thesaurus ' ■ * 

APA Thesaurus (Psychology Abstracts Data Base) . 

ERIC Thesaurus 

, • BIOSIS Master Index Authority File 

. IRON DATA CENTBl ThesaUfus ^ 

^ MeSH Vocabulary > ° ' 

^ Management Contents Thesaurus 

While several of the format conversions iiere s'traightforward, three of the 
I vocabularies deserve special nentron. - >^ 

The IRON DATA CENTER thesaurus is 1nterest1*ig since U is 
trilingual, providing Span 1el-Por^gese-EngJ1$h* translation. This vocabulary 
has demonstrated the capability of VSS to perform such translation. 

The BIOSIS Master Index Authority File proved interesting since It^- 
did not resenile *a conrentlwial thesaurOs ^structure. In general, the BIOSIS 
file Is free text and the Authority File was set 19. to indicate how. to best 
search a topic. The following exanples Indicate how the Authorl^ File was 
restructured to follow a conventional thesaurus structurci 



MIAF ENTRY . THESAURUS STRlKfrURE 



AARDVARK AARDVARK 
KW: AARDVARK (20) 

If KW matches, set up a valid lead term. 

• W ;. y ^ 

, ' ABATEMENT ABATE 

KW: ABATES (170) ABATEMENT use ABATE 
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If KU does not match, set \sp USE reference. Object of USE- Is set up 
{as valid lead terai. Hote that truncation Indicator* is drbpped. While this Is 
not desirable far several reasons. It 'is necessary %o allow switching to other 
vocabularies. " 

ACCESS ' • ACCESS use 

KM: ACCESS (ISO) ACCESSI$ (210) ACCESS, ACCESSI ORed 

. > ACCESSI use 

* * ACCESS, ACCESSI ORed 

If Bwltiple words in KM field, set up. IKE reference to scope note. 

ACETYLCHOLINE ACETYL . - ^ - 

TCW: ACETYL & CWa.IIIE ACETYLCW^-INE use 

AOTYL .m^^jmJinE 

^ ' • CHOLINE 



)f -Al 



set up USE reference, to sp«;Jal "AND" construct. Jtoke 
sure each part of^ "AND" ccmstnict Is a valid lead terw. ^ 
[ SEE ALSO entries w?re ^ijolated as related t|r»s. /These a>^ not used 

in the curre>it version of but have been^xaj^tured for possible future 
inclusion. 

The Concept Codes and Blosysteaatic Codes were sapped intd their 
headings. ^ . ^ * 

. BC: BC07600 use 

MYCOPLASMATALES MYC0PLA9WTALES 
1969-78: BC07600 C2630) 

In addition, a second vocabulary (called the BCCC vocabulary) iwis 
defined which saps headings into the Concept Codes and Bipsystematic Codes. 



BC: . - MYCOPgVSMATALES use,' 

MYCOPLASHATALES BC07600 
.1969-78: BC07600 (2630) 

« ■ ' 

This approach was chosen. with the ca)ES since any one of several 
relationships may exis^ between a lead terra and the associated codes. 
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Therefore, we defined relatlonshi^^between Codes and Headings rather than 
between Codes and lead terws. 

. Although several other minor points were addressed, the above 
exanpl£s Indicate bon the BIOSIS Authority File w;a$ restructured for VSS. 

TTie final vocabulary iihlch deserves special mention Is the NeSH 
vocabulary. .Die standard NeSH vocabulary was converted In a fairly 
straightforward manner. However, In addition to the standard vocabulary 
records, we received a large file Of chemical records from NLM. 

^ / These rteords^ contain mappings among NeSH headings, CA registry 
numbers, and chemical substance names. With this- Information and work similar 
to thet performed for BIOSIS, three NeSH vocabularies were created. The first 
was the standard Ne^ vocabulary with the addition that CA registry Ttunbers and 
chemical substarwaie naaes were mapped |» the appr^riate fteSH heading. In the ^ 
second file the substance names were mapped to the proper CA registry nunfeer, 
and* In the third file ^ #«g1stry mmtoers were mapped to the substance names. 
This allowed powerful switching for chemical topics. 

The current VtJS Input format is shown In Figure A-1. 
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FieUiS A-l. VSS INPUT Fomi 



• ^ RECORD lAYOOT 

Position Content. . 

1 Relationship Code (Alphanumeric, See Belo«r> 

2 Blank 

3-4 Line Sequence , Number — for Continuation Cards. . 

(e.g., 01, 02) 

5 ' Blank " . 

6 ./ ^ Vocabulary Source Code (Alphabetic; Arbitrarily^' 

Assigned) 

7 Bl^nk - • 
8-^7 Term 

68-80 Blank . , . .* 



Helationship ' * . ' . ' . ' 

Code . Relationship ^ 

0 LEAD (or MAIN) TERM 

^ 1 Sa>PE NOTE * V . 

2 , USB 

3 . 

4 OSPD-FOR (OF) . . > 

5 DF+ .. .» 

6 SPECIAL SCOP^ NOTE 

'7 'BROADER TERM/ (BT) . • 

8 NARR0I7ER TERM (NT) 
9 

, A SEEN-FROM (SP) 

B " RELATED TERM (RT) as de&ii%nated by supplier 

C Subject Category am designated by. supplier 

D Suggested NT as designated by s\%>plier 

£ X Suggested BT as designated .by supplier 

F ' suggested ^T as designated by st:^}plier 

G Array KIT as designated by supplier 

H ^ . Array BT a^- designated by supplier 

' . I * * * Array ET as, designated by supplier 

, J Top terns. , • , 

K Frequency count 
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APfOPiX 8 ' ^ 

/ ■•/■.■■ • . • 

' • ' , fSS FILE-«iILPMft MPnigS m REOKD $TOCT«ES 

[ Ir^ut to «ie ^Voc^lajry Switching ^jfsta?! fVSS) file building 
op«*at1on consists of ill preprocessed voci^ulvry records- In.VSS foraat. (SeJ 
Figures 1 & 2 for exwples of vocabularies In VSS fonwt.) Lead ten» must be ^ 
In alphabetic order. Vli«fi a lead tera weared In sore than one vocabulary, a 
separate entry was created for eadi occurrence. These redundant occurrences . 
were further sortisd by vocabulary source code. . ' 

^ Following each lead .tera entry there ^appearetf various relational 
entries, such as USE, laulttple m. (used for), scope-m>te. BT (bfoaiter 
}i,,terj), irr (narrower tera) m4 RT (related tera). the relational entries 
V. iSrfi that Is, a BT wtry wider or» lead tera woUld be aatqhefl . 

by a corresponding HT ent»^ of the lead tera under Its broader tera. 
Consequently, nearly half of the entries In the li^iit (one *itry of each 
reclprocatefl pair) were not essMtlal to build the VocAularif files. For 
process.1ng efficiency, - both BT and HT entries were accepted as th^^ere 
encountered, togettier with USE and special scope iw^ «itr1e$. 

To fora the to the tara, phrase, stea,, and word files, alV 
«itr1es 'were edited by a tera standardliatlcm routine ,to ellalnate ainor 
• variations due to puiwtuatlon and spacing. This routine converted all 
characters exc^ letters-^ nweerals to spaces,^ 1ii«i r^ed all wiltlple 
^spaces to single spaces, no further transforfflat1on''(sifch as singularlzatlon'or 
removal of prepositions) was ^e^lo/ed* Teras that , wouli: eventually end up In 
the conc^t file were not edited and are referred to as unsanltlzed terns. " • 
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.Entries consisting of teoil, relational qode, and vo<?abulary code, 
were processed sequentially. Records for lead tera entries were built up In a 
working area as the various relational entries were processed, and then were 
written, to a file keyed by standardized tera. Records for relational entries 
were created or updated and written to the file before processing ^tffe next 
entry. * . ' 
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mnUKE^ lOCABUURY RECORDS 
MeSN" HESMBBS (IfeSHr) 



ABC 



tem ^ 



9 g R I P I Nn-gtygt^PIOUg ■■ : . ' _ 

C 1 R l»l>8.-HYPRQXy«»*S*fc*7^TEtRAHETHrL>lM>g^BgM2hPrRAIi«l ^.r^ttfrO X . 

0 2 R ONE 

g 1 R 13277»7&-» : ^ • ■ 

0 1 R (<^)-N-^CYCLOHEXY^-Nt«{4-f3-(!X#l*0XHETHnETHYL)AflXNa)-^HIYDR 

0. 2 R OX YPROPQX YIP'HEWYHUR EA • l. . 



2 1 R 57460-4X-0 

0 I R C4'>)-l^ACET0XY^3^6ir6A>7»8*9#10#XQA-0CTA HY0Rt^9 MYDR0XY-^B6t 

0 2 R A-HETHYL-3-l5t-FHEIfYL-2t-PEMTYlOXY>PHEHAHT«RXOX»fE MYOROCMIOR 

J? ? ,» m - — ■ 



2 

0 



1 R 72028-54-t ^ 

1 (♦/>)>kLPHA-HYDRA2XNfl-3>4>eXHYDRQ*>2Il MHXSOQUlNOL XHE^UTAIiOIC 
C 2 R ACXP > 

^ S84§9>?2-Q : ■ • ■ ■ ■ , : 

0 1 R ('♦/-)-HflH0AR6E«0HXM£ . 

2 X R 39013*26'-8 ' ■ ^ ; . ^ ' 

0 1 R (♦/-)-l-Ca#X-OXnETHYHN^'IANXNO)*$-(2-rC2-fROPtl*YLOXY)FHENO 

0 2 R XY)-2-f>ROPAN0Lj HYDROCHLORXDE 

2 I R 36902-82-6 

0 1 R C»/-)-l*q ijX- DIHETHYLETHYtlAHXWQIjfe-f4*l»^^TRXFL i PR0ffEtHY H 

0 2 R -1H-1«X4)A20L-2-1 

2 X R 6296 0-7S-2 



*Y L } FMEflOX Y 1 -2*F ROFMIOL 



J 



A - relational code 
B - card nunber 
€ - vocabulary code 



I&latl^l Code: 

6 - Lead Tem 
2 « Use 
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A I c 



0 
2 
0 
2 



IWSTITUT P UR EISEHWUTT EMWifiM 
TECWIXCAL (fflXV QF AACHEN 

•EUROPE - 



C 
4 



4 

4 



7 
7 



0 

0 



0 

6 



0 
0 



AB SAHCO 

AB COLP-BOUND PELLETS 



COLD- BC^MD PELLETS 
AB ST HILJOTEKWIK 



abbattista; p. 

ABBOTT. A>P. 



ABE# H. 



ABEf Y* 
ABEL. 0. 



C 



ABEX ccmf • 

ABLANPAlTlEWTfl 



1 
2 



(SPA) 
SOPTENiHft 



ABN AHNUAL COMMESS 



0 
0 



ABRAMAH# iK*PV 
ABPAHS^ H,B> 



\\\ 



ABRASXON INDEX 
CSfS^IP^qmON 



0 

1 



7 



llAHi 



lUDfCE DE AttASilO-faRT 
INOICE DC A 0RAS j¥I0Afl^SM 



PHYSICAL $PECIF|iCATlONS 
SPECIFICATJOWS 



RelatloitR^I Codes: 



0 " Lead Tersi 

1 • Scope Note 

2 a Use 

4 - Used - For ' 
7 ■ Ariray Narrower Terp 
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As e«ch new entry was accepted (except for imiUiple US£ entries) the 
keyed file was accessed to detenilne whether or not the terra had been 
encountered previously. If It had leen the existing record for that term wis, 
read In -and stw^ for updating. If not, a new record was created and a 
■concept m«i>er" was Assigned for Inclusion Hn the record. The concept nueber. 
aerely the next aV^ailable integer, was assist whether or not the tern was a 
valid concept, subsequent references to the ter« used the concept nurter 
rather than the text of the ters. These references are described el seirfiere In 
the discussion of conc^t-mster cells. 

A special procedure was alloyed In the case of nwltlple USE 
entries. Instead of accessing the Individual terms designated as fultlple IKE, 
they were used to bull d a synthetic tens conslstfK^ of the several USE tenas, 
separated by the operator ■.AW).". »ien the last ^ult'ipl^-UeSE reference under 
a lead term had ^'processed In this sariner, the co^lned expression was 
treated nuch the saiae as a slrgle-term conc^t. That Is a series of Input 
entries consisting of : 



:./v 



TERM A 

US&1|ftM B 
USE TERM C 



would be treated as^ 



. TERM A * • 

i , USE TE»1 B .AWL. TERM C 

and the- expression TEIW B i,AHO. TERM C wwild have an assl^ed concept nuipber 
apart froi'the conc^t ni^^rs nislgn^ to TERM A, T^ $, and TERM. C. . . 

> In a like wner^ «M»:f^^ scoj^ note expresslwis were handled as If 
they were single terw cbnc^ts. Tlius; iii art; witrys^ • .: - . . " 

■ • ■ jmi D . • \ . • 

• • • • \'f\ ■■.tisrTERM. E OR' TERM F; ' • • ^ • 
th^ expV^sslon rtpil/E OR TQ» F was ponslder^ a c^ '. ■ , 

the actfpfa descried abov^ created a t«|>prary file c<»ita1n1r^ one 
record for ewh ^Ique l^ad ienn^ and a nwiber of ^ddftjonal records for the 
special scf^e m>te 'of swltlple US^^^^'c consisted of one 

tbnc^t niiber piircei (described el sewbere'). first concept nu^r cell 
contained the concept nurtjer for the 'teria .and flags Indicating those 
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vocabularies i^ere the terw was valid. If ^e term was not valid, a second 
concept niffltoer cell was used to Identify^ the valid term (the *USE cross 

reference). ^ * 

Using the saae standardized ters as a key and following the sane 

processing algorltfw, a second tei^wrary file was buHt containing one record 

for each unlqiw l€ad t«n8, multiple -use tera, and special scope note tern. 

Each M record contained a unique unsanltlzed term for each applicable 

vocabulary. , 

f. ' ■ .... ■ t ■ ' 

• • • » 

PASS 2v / \ . » . 

The files created In PASS 1, tenporary field and unsanltlzed 

♦ . - 

term filei were used as Input to produce a TERM f^le and a CONCEPT file. Each 
record In the teB|>orary*f1Te was processed In sequence. Lead terms, whether or 
not they were valid ewic^ts, were -uswl as keys to term .file records 
containing* Idad-term and USE (both single • and^ltlple) cells in a concept- 
nip^r-parcel-record. A. concept file recorj was also created 'for valid 
concepts only^ for this file, iihe cwS^ h^^^^ 

record Includes vocabulary Vlags ftfr lead-term usage, unsanltlzed terms of the 
concit for $acft vocabulary, and. If appn^rlate, a conc^t-iiuirtjer parcel 
containing BT and NT cells from the CW> of the temporary file. U 

PA^ES 3 and WimSP ' * • 

Of ' ' ' • - ■ • ■' 

PASS 2 also generated a relatively small nwrt>er of related concepts, 

cal'led co-related terms (CRT) In VSS. If f^r a Siven term TERM A In the term 
file one vocabulary said USE TERM B and another vocabulal^y said USE TERM C, 
then a "co-related term" relationship was assinned between TERM B and TEW C. 
Thus, for any vocabulary containing both TERM B and TERM C as valid concepts, 
each wwJld reference tte other as a* CRT. If for a third vocabulary TERM A was 
valid, the CRT relationship would exist between TERM A, TEIW B, and TEfW C. 
PASS 3 adds these co-related terms to the CONCEPT file. 

MAKEUP also uses the TERM file generated by PASS 2. It finds all USE 
references In the TERM file and reciprocates the relationship by creating USED 



127 



FOR references. As«1n PASS the.CCmPT file Is then upd&ed with the USED* 
FOR tents of this relationship. 

< 

• ■ ♦ » • • • 

PASSES 4, 4P. 4S/4tf ^ ' 

• • • ■ f ^ . ^. . 

The tens f^le was used as Input to build three additional flles-^ 
PHRASE, STEM and «(»D ^lles. 'Each key In the ^terw^lle was processed In 
sequme» with" the toay being transforaed Into a sj^Med phrase, a sef1eS|0f 
Individual w^s. md a series of Indlvlduai'ttaBS.' The algorithm used ^^n 
these tpansfonaatlons .was Identical to that- used In the Logic module of ihe 
switching system.. ' - j 

' For. each key grated, that is, for each phrase, each word, and 
eadi st«i. a transaction was Issued that contained the key and one cell from 
the term file conceptHliwber parcel. Th1i|. was <tone f or each' cell in the.CI«>. 
The reason for breakli^ 19 the term file, recorcte In this manner was the teter 
need to merge cells uwter (m key that came originally fnp many dlffeihent 
term-file records. For exaiple. two terms In the term file might be 
ELECTRICITY, and tLECTRDRS^ Cells for, both of these terms would be found In one 
record in the stem file under trie key ELECTR. • 

The three sets of trw$i«t1ons generated in 4 (phrase, word, 
and stem) w^ processed In a similar manner.. First, all transactions mre 
. sorted;' with the primary sort tm the key. Then thay were processed in orcter, 
elliier adding new cmce^t-miBb&r cells or adding voctijulary fUgs to existing 
concept-nuai>er cells. (Jhi^e that til? coiwept-mjmber cell is comprised* of a 
concept number, a relationship code, artd a yocitf>ulary-flag subcell. Thus if 
two cells have the same cimc^t mmiter and the same relationship code, their 
vocabulary-flag subcel Is can be merged.) 
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^ Mm 
Usage 

Table Description 



• (mm wmm ?mxi TfBUE . 

The CUP fonaat Is used 'in the CWCEPT, TERM. WORD, STEM, and 
fmst files. In all bat the cmEPT file tRe- CUP cosprlses 
.the oitlre r0cord foraat. . * , ^ ^ 

« ■ . ^ * 

The CUP Is a data striicture for the r^resentatlon of ,conc^t 
mul^ers and related Infofiiatlon* 



cpgRHL CMP mmr 



Cell 
1 

2 • • • • IH*X 



Description 

f 

C^ cell count (N) 

Conc^t mnber cells (see' 
below). 



cmsm wamst cell rmm 



.,.,1 ...immr.. m.n.rT^.^ Trfll Ih lei^th plus flllBT ^ rcQulred and It 

occupies one or two cowputer words. The first \m subcells are left-^^^J^^^ 
first word of the cell and the, last two subcells are rlght^Just^lfled In the last word of 
the cell. , 



of Bits 



24 



(Variable) 



DfescarlPtlOii . v . 

ConcMt fliffl*er fhe ctmc^t nurtber IS the unique' 1dent11^1er 

assigned to each Idmttlflable. VALID (postable) 

Index terp/thesauru^ tens In the concept. file. It 

can be Used' as the key to the coYic^t file. ^ 



Concept Type 



\ 

Filler 

Discretionary 



The concept type indicates the relationship of the 
con< 
the 



conc^ nifflijer to the key of the record containing 



0 » use for 

1 ■ co-related term 
Z » narrbwe^' terw . 



3 » broader term 

4 « related term 



This ^leld way be Ulsed by different modules Jfor any 
purpose desired. Care imist be taken however to 
Insure that no conflict arises In Its usage. 
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Ilmbmr €f i1t$ 

24 , 



VBT - 
(Vocabulary. 

bit table)' 



■ C~ ■ 

PascrlDtlon ^ • . 

THIS table ctenotes Mhat vocabulary(s) /thesaurus 
this collect Is valid In. The presence of a vo- 
cabulary Is Indicated by the presence of a "1 * bit 
In the The vocabularies bits are niwNred 
froB left to right frow 1 to 24. The vocabulary _ 
code corresponding to sny given position/bit mjf be 
retrieved- frow the VCT» vocabulary table, »<here the 
posttlon (1 to 24\ Is the indek to the VCT. 
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CaKXPTFILE * 

The Concept File Is keyed by concept nuoiber and contains (1) the 
corresponding unsanltlzed term 4$ It appears In each vocabulary, . and (2)» a 
comjept nuisber parcel containing concept niartber cells that point to co- 
related, narroK, broader, and related concepts; 

CaiCEPT FIU bwCt 

Orgaiflztlon: Src'{ symbol 1c keyed file). 
• Binary records. . 

Randon access. 

Key: Concept maBber (24 b,1t unsigned maferic): 

Record; The record contains the unsanftlzed tekt of the concept for valid 
vocabulary ter«s along with vocabulary flags Indicating' In imlch 
vocabularies the un$an1t1z«l ccmcept tera Is found. Also 
Included are pointers that link a concept to other concepts in 
the concept file. Ttie identified relatlwishlps are; 

cdi-related term-CRT ' ' • 
piarrwef ters^ffT , , • 

broader ter^BT 
related tem-RT 
use -tera ^ 



Cell 


LeAflth' 


1 


1 word 


2 


1 iwrd 


3 n 


Variable, up 




1 mrd 


n+2 0 


S^arlable, up 




to 5 iwrds 


0+1 


1 word 


0+2 


1 *«rd ^ 



PescrlptlOB 

Record Information cell (sefe belo^«) 
Concept Information cell (see below) 
Unsanltlzed term. This Is the orglnal lead term, to 
'5 wordsBoolean conjunction of lead terms, jor special 
scope note when regarded as a valid concept 
Saalr as cel l 2 
Same as cell 3^ 



Niflflber of cells to follow • . 

Concept mnber cell (CMC) as (tescribed elsewhere. 

for each CRT, NT, BT. RT, or use term 



One 



X31 



9-10 



mamnm cell fumat 



of Bits 



6 
6 

22 
2 



24 



This* cell U 60 bits lif length and occupies one coaputer word. 

Description 

Niinber of unsanltlzed tenss stored 



Coun€ 
Pointer 
Finer 
Flag 



VBT 



Word position of the cell preceding the CMC 
relative to the record Information cell 



This flag" Is set to non-iero when the concept is 
not a lead tetm^ thus not a TERM file Vey: 

0 > conc^t Is lead tern 

1 « concept Is B<K>lean expression 

2 " concept is sc(^ note 

Vocabulary bit table Indicating those Vocabularies 
In which the unsanltlzed term Is found 



of Bits 



6 

30 
24 



Count 

Filler 

VBT 



FT nfORmnow ceil fowi^t 

PiscrlPtlon 

.lawyer of Mrds to holid unsanltlzed term 



Vocabulary ^t table Indicating those vocabularies 
In which the unsanltlzed term Is found 
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VSSfOCmUMIES 



i 



Vocabnlvy Vocdmlanr 
code 



A DOE Thesaurus 

B ABI Infoni 

C CA Com:ept Edit 

D ERIC ^ - . 

E SHE Vocid)uUry (EI) 

F Iron pata tojter 

G • * M«iage9«it /Contents 

H BIOSIS 

I INSPEC Thesaurus 

n mbSh \ 

N ' ' NASA Thesaurus 

P PsycA Abstracts (APA) 

R * nm' 

.S NbSH" 

T BIOSIS^ 
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vss oKRAniK nsnucnoNS 



o 
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vss iisnucTin 

leOKLET 



Nirck 4, 1983 



f 



MTTEULE 
ColHte LabMtoHes 

Cohvtas. Ohio 4^201 
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* ■ . . 

Battelle Is condtietlilg research imd«r • Niftloiial Science Fotmdatlon 
grant (ISX-8111497)^ evaluate a coBputer-asslsted onllYi^.ieairc^h aid called \ 
the Vocabulary Switching Systea (V$S). ^ • . s . 

^ Your participation In th1s^tii4y Is entirely voluntary. If you 
decide to participate, ywir 1«tent1ty wl^l be kept C(mf1(^t1a1 In all report- 
ing activities and/or publications that result froa this study. ResuTts^iHIl 
be aggregated and analyzed In si»h a way as to pi:event tracing to any 
Individual or coqiaoy« ' , 

Hoi^r, we uMil^ like to acimoMledge all participating organiza- 
tions by n«»'1n k preface In^^r^Ylnal re^oft« This. Is our jMty of thanlcing 
^you 'and letting readers fam Just tow broad, the respoMtent base MS. 

Please read and answer Ihe first 10 0iest1(ms of the VSS Evaluattoi^ 
Fora before you cgnRsnce an actual online sasslon. Also. It Is strongly 
recMsnded that you read the ranalnter i»f the VSS Evaiuatlm fons before 
1og|n so that you beeoae acqt^lnted trftk the types of (^stlois to Jw 
ansiiered during and after ttm test peHdd. It Is liaportant that Question' 
11 be coBpleted during M test period after each mrch Is coopleted. 
, (^stlofis"^ through 17 should be ^oapleted at the conclusion of your test 
period. Return the completed evaluatfcm fons fn the envelope provided. 

You are not/confined to rwnlng 10 s^rctes-on VSS as provided In 
Question 11. * If you can perfona aore than 10 searches during your test 
period, please feel frte to do so.. Just photocopy .one of the fonas and " 
ctmtinuB to nuober yotfr searches. ^ 

Your Bp^t Ij^rtant guideline Is to try- to stay within* a total of 
about 2 hours of conyct tiae per participant during the assipied test 
period. You can keep a running total of your own tletf by noting the connect 
tiae when you logout. 

. Since no foraal training Is required. Just 'read the login 
Instructions, and other aaterlats enclosed In thls^ packet and begin your 
evaluation during your assigned week. * . 
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The login procedure is tedious; we apologize In advance. If TYMNET 
responses are slow. It my be due to TYMNET voluise or Battelle computer usage. 
Our computer usage tendcto peak at about 10 a.«. and again at 3 p. a. EST. * 
Your response tiae will be better If ^u avoid these peai periods. 
Occasionally you can Inprove response tint by dialing us direct (614) 424-5450. 
However, the cost of direct dial Will be billed to you, whereas a TYMNET call 
will not. 

If you have any trouble with login or a systea glitch, call us » 
collect on one of our two HOT LINE ntobers: ' 

(614) 424-7843 (Bob Niehoff ) 
/ . (614) 424-6386 (Helen Pestel) ' 

Your packet contains: . - 

e Cover letter 

e laportant Instructions 

e VSS Login Instructions 

• YSS Switching Features 
- YSS Tens Types 

. • VSS Evaluation Foras 

e Exai^les of Subject Switching 

e Return Envelope. ^ 

Please return: 

• ♦ 1 • ■ 

e Evaluation fonB(s) as soon as possible after your test 
^ — ^jjertod. 

t Printouts of your VSS online session If you used a print 
tertelnal^durlng the evaluation period (optional). 




VSS uses Battefle's BASIS, a data naiia^Bnit systea for 
Inforaailon and data storage and retrieval. 
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- ySS LOfilli IRSTWICTIOilS 

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

(1) Set terminal or B»d«i to half duplex unless you have DEC equipment. 
For DEC equlpaient, set "local" sode to On. 

(2) Dial a TYMNET ntad>er Iri your {irea. . 

. [Please Type%Your T^ralnal Identifier j * 

(3) You Entet; your tenajnal Identifier from table tielo^: 



I4Mtlf 1«r 








A 

• 


ASCII 




tatoMl CMniuttri 


1 


Affit 




All UnHMls 


C 


Ascn 


30fe|pt 


XipKt prlnfcM 

• 


8 


Asen 




All tmlMis 


E 

* 


Aseu 




Itanni printers 

• • 




ASCII 


ISepf in^ ' 
JBcps 9Ut 


0 HTTA t«n|rtMlt 

« 




AScn 




orlstart, 
a.C« TflnrfoK 


1- 






i^litf Is prlstvs 




BK3/ 




S«1tetr1e-ta!pt 
tvoriMls 
(«.«., 2741) 



] Please' Log In; 



(4) You Enter: BATTELLE [Cft]** * 



p.nn 
BCL 9 Is Online 
Itestlnatlon? 



Systesi response. 



ICR] - Carriage return 



€-5 



4 * . . 



(S) You Eritcrr CDC 



1 



W . You Enter; [CR] '^^s urrc-l 1 fasten T^sporj<ls 




■ '. ■ I , ■ ■ ■■ . I • ) ■ • ■ •' • ; i . * 



togiN 



(7) You Enter: L061W. : : 4 .>ISf^.SUP.N 



(8) 



You Enter; ETL,7777 



Coopsnd** 



(9) You Enter: BAStS.WW.VSSi: 



ttelCOBQ to V$S.'.. 

VSS 'contains* 



(10) To logout, select END THIS SESSION fros the appropriate VSS menu 



Cosnand- 



(11) You Enter: LOGOUT 



DO m 



Request broader or narrowr tem^^BT/l 
vocabularies; these vocabularies oe- not 
files. 



') for Business or Life Science 
ise BT/NT relationships In their 



Request Item 6 from the menu of swItchWig options unless you are prepared 
to read the attachn^nt to learn bow to i^t up your own switching strategy 
Even then, this option may be confusing. 



Use the browse feature frequently because^ 1t has the most switching capa- 
bility of all the options. However, It does not have BT/NT. 

Send us a^rlntout of your online sessions If you use a print terminal. 

f ■ * 

ft 

Set the number of terms to be displayed per vocabulary to relatively high 
values (5 to 10) to generate sufficient output. 
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(1) You cannot perfbra sifltchlng across vocabulary sets slnultanedusly. 
' Instead, you mat work witli one set at a tine. If you nant to 

perfom siHtcMng In anotlier set, seleet EMTTHIS ^ESSKM fron the 
appropriate VSS wmu. Mien the systea profits you witfi COMNAfti)-. 
If you enter BASXS.RUN.VSS Instead of LOfiOrT* ymi «d11t« given • 
the opportunity to select another vocabulary ^t (teflned by VS$ 
Id thout going throi^ the conplete login $e<^fence, 

- , ■ • • ■ ' ■* 

IkMevert you can change vocabulartes trtthln Life and Physical 
Sclente sets by selecting this option fron the appr(H»r1ate VSS 
nenu. " 

(2) The thesauri used In VSS are nob several yeav^^ld. 

C3) VSS contains the folloiring vocabuTaHes: ^ 
business 

ABI - ABI Infom Thesaurus 

* ■ ■ . ^. ■ , ' 

Contents - Nsnagetent Contents Thesaurus 

- •'■/'/ 
Social Science ? ' 

*EKIC ERIC Thesaurus 

Psych ' 'f ' . 

^ • Abs - Psychological Abstracts Thesaurus 

f Life. Science 

BIOSIS - The Ptaster Imtex AuthoHty File (MIAF)^fron BIOSIS. 

In the VSS version of this vocabulary, BC (Biological 
Codes) and CC (Cracept Codes) are Invalid. 

'BlOSIS-f - A special version, of BIOSIS In which CC (concept, codes) 
are valid but the ccmcepts are Invalid* 

NeSH - Ihe aedlcal subj^t headings of the PCDLINE systm 
used at tf^ National Library of Medicine. 

NeSH*R - A special 'subset of NeSH In which substance nanes are 
Invalid but ttelr equivalent chenlcal registry nuRbers 
are valid. 

NbSH-S - The Inverse of NeSH-R 

CA - Chenlcal Abstracts Concept Edit File 
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Physical Science 



C-7 



CA 

DOE 

EI 

INSPEC 

im 

NASA 



- Ch«Blcal Abstracts Concept Edit File 

- DepartsMot of Energy Thesaurus * . 

- Subject Headings .for Engineering, froB Engineering 
Inforaatlm Inc. (fomerly Engineering Index) 

- INSPEC Thesaurus 

- A trilingual thesaurui on Iron netallurgy. Languages 
fnclude English. Spanls^^ and Portuguese. 

^ \ ' ■ * • ^ ■ " 

- NASA Thesuawis . . 



VSS yCKABULARIES 



Thesaurus Relationship 



Vocabularies # 
— ^ 

Buslnness 

Adi Inform 
- Mgnt. Contents 

Social Science 

ERIC 

Psy..Abs. 

Life Science 

8I0SIS 
BJOSIS-C 
CA 
« NeSH' 
NeSH-R 
MeSH-S 



Lead 
Term 



Use 



Special 
Scope 
Notes 



BT/ 
NT 



RT 



PhAlcal 

CA . 
OOE 
EI * 
INSPEC 

. mn 

NASA. 



Science 



9* 
1 



8 
6 



17 
4" 
14 
74 
42 
92 



. -14 
22 
12 

..^ 17 
^ 16 



1 
0 



3 
t 



7 

0' 
3 

36 
28 
31 



3 
5 
3 

-.-4 
4 

19 



0 
0 



0 
0 



0 

0 
63 
0 
0 



0 
0 

<1 

0 
0 

<1 



0 
0 



5 
6 




0 

a 

0 
0 
0 



0 

30 
0* 
5 

16 
78 



6 

0 



36 

7 



9 
0 
0 
1 
0 
0 



b 

27 

0 
15 

5 

132 



(4) The pronpt s^nbol In USIS U "7". 

\ '■ ■ ■'. • 

(5) The Inverted file record length Is 50 characters, so If a YSS 

vocabulary entry 5s truncated, consult the printed yocabulary to 
Identify the cflsplete tens. 



(6) If OI£D aiH^rs et the end of a YSS tern, this mns that the terns 
proceeding It shoultf be OR'ed together In BooTean fashion: 

(7) If '.AUD. appears anyiihere In a V$$ tens, t^ terns should be 
AND*ed^ In Boeledh fashion. An m without %he pet^ods Is a 

. IgraMtlcal ANO» not a Boolean AND. 
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• ¥SS StrttdtliKi Features * » * 

1., Sytiowys t 

This feature Invokes a look-up across all vocabularies vrfthln a 
VSS Biodule for occurrences of: 

(a) Your tern ' . 

(b) Synonyii» 

(c) Corel ated 

Synonyins are defined as all USE, USED FOR* SEE, and SEEN FROM cross 
references associated with your tern. Correlated terns are those terns 
which are related by virtue of a cooBon ancestor, e.g., B Is co- related' 
to C If the following thesaurus construction Is Identified by VSS: 
Vocabulary I: A use B 

Vocabulary II: A use C | . 

Two labels are used at output to Identify successful sirrah- 
Ing; YOUR TERM, and SYN(MYM4-, shown under the col mm heading "Tem Type". 
SYNCHVYMi' Is used to label the class of output consisting of both synonyns 
and correlated terns. ^ 

This feature autonatlcally invokes nany types of VSS switching \ 
options In a pre-defined order, the. options, called tern types at out- 
put, used by Browse and their order are: 

Tern Tjfpe 
your tern 

synonyn+ . 
rel phras 
related 

* wd natch 

stn mtch 
adj-lead 
adj-word 



ERIC 



* The amount of switching that takes pUce* within any VSS feature is 
United by the nunber of terns to be displayed per ^vocabulary ^ 
Therefore, a switching feature like BROMSE nay never fully execute 
all of its pre-defined options if the nus^er of terns to be displayed 
per voca.bulary Is set too low. ^ - 
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Refer to the next section for further definition 'of terai types (or ~ 
snitching options. If VSS does not p^rfom all the sidtching (defined 
above. Increase the niBSber of teros to be dIsplayiHl per vocabulary 
and resufaslt your search tersi. Iterate ^th1s process until ^u are 
satisfied or VSS sliq^ly cannot produce Bore output. 



This feature Invokes a search of VSS files for all -narrower 
terns (KT) associated trith "your tens*. 



4. Wrmiier imm 

This feature Invpkes a search of VSS files for.all broader 
terns (BT) associated with "your ter»". 

Sm Wii iCMBi'/^mrfcy Term 



This feature conblnes options 3 and 4. In thatlrder. 
6. Other CtisaMiBflMWi) 



r 

toNbull 




This feature allows you to^bulld your own sw1tchKig[^ options in 
any coablnatlon and order desired, m SHOUlD^COIimT "HELP" 
CmmO TO PECONE ACQUAINTED MITH VSS CONNAN^S BEF(mE USING THIS FEAtUl^. 
SIMPLY ENTER HELP. 

When you select this feature, you will be given a Isnu of VSS 
switching options In "systeB-ese"^ ' ^ 



# 
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The "systei»-ese" fienu apMrs {is follons: 




AVAILABLE ^ITCHIHS OPTIffiS INCLUI^: 

05 - TERN FILE ACCESS - ACCEPT LEAD * SMITCHABU TERNS 

06 * 5TEN n«ASE/IL£ - AC^T LEAD i- SNITCHABU TERNS 

07 - Wm FIU aKeSS ^ ipTPCT) NATCH RE(9JIRED 

08 STEN FILE ACCE^ - (StFCT) mm R&)UIRED 

09 CMEPT FILE /^SS - ACCEPT REUTED TE»6 ONLY 

10 - COt^EPT f ILE ACCESS - ACCEPT NARRCM TEMIS ONLY 
11- COmXPT FliELACCESS - ACCEPT ^t(^R TERMS ONLY v 

V 12 - TCWI F|LtTid&» - LIST (2nCTA0J) ADJACENT TE»!S 
13 - WORD F!L£ ACCESS - LIST (2* SETAOd) AOJACERT NORDS 

- 19 - ACCEPT LEAD -i- NDLTIPLE USE * NULTfPLE O^FOR 
21 - I^LATED .TEIWS * 

Jem types associated trith these optfoiks are: 



SMtchfas 




*«o«* 


Tetli Tm 


05 


temfsyn 


06 


rel ' phrase 


07 


ltd aatch 


08 


stm aatch 


09 


Corel ated 


10 


narrow 


11 


broad 


12 


adj^lead 


13 


adj-word 


•19 . 


synonyp 


,21 


related 



To use this feature, sloply enter the nusber corresponding to each option 

desired; one nuiter per prospt (the proapt sjfa^l Is 7 In this conputer 

systen) untl\ you have entered all selectlcms deslr^. The selection 

process Is concluded tdth the word STORE. A typical session night look 

like this:* 

, - 75 

7 19 . • 

7 9 
7 10 

• 7 7 

7STQRE 

. Enter a search tens for coonand • 
SETPCT 
Enter an Integer value 
'« . ' 7 66 

Enter a search tern for coasand 

YOCCNT . 
Enter an Integer value 
7 5 
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YSS Tewi Types 

iU adj lead stand%for adjacent tern. Adjacent term Is sloqsly a 
browse of the Inverted f11« for those terns which &rt adjacent 
to the user-entered tern. The nu9d>er of adjacent terns displayed 
.. • . before and after the user-entered tern (the window) Is 7 by default. 
If larger or narrower windows are' desired, the user oust define 
then via the SETADJ connand In VSS. The actual window width *d1s- 
played Is also Influended by the nia^r of terns to be displayed 
per vocabulary, because .tte vocabulary^ count specified by .the user 
take^ precedence. This label Is used In con Junctton with switching 
option 12 In the user-defined node of VSS. 

' ■ . . • . 

'(^) adjHiord Stands for adjgcent word. Adjacent word Is Identical to 

adjacent- tern except that Individual words are extracted frpn the 

user-entered tern and used to "bhjwse the VSS Inverted word file. 

This label Is use^ln conjunction with switctijing option 13 in the 
. • user-defined node of VSS. * , 

. broad stands for broader terns (If). This latile Is Osed to Identify 
all the broader teni» ftsoclated with "your tern". This option 
. works In a United sethse In the DOE Thesaurus; It only produces . 
broader terns at the BT 1 level fn^ the DOE Thesaurus. This 
label corresponds to switching option 11 In the user defined node of 
. VSS. ^ 

I ■ ' 

(4)' corelated stands for the co-related tern. This label Identifies 
those terns which are related by v.1rtue of a coanmn ancestor, e.g., 
B Is CO- related to C If the following tliesaurus construction Is- 
- Identified by VSS: • 

. * • Vocabulary I: A use B 

Vocabulary II: A use C 

' This label Is used In conjunction with switching option 09 In the , 

user-defined node of VSS. Note that when a cp-related relationship 
Is Identified In the Browse feature of VSS the output Is simply, 
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labeled- synonjms. This labeling anomaly Is due to the fact th^t - 
under the BfOM&e feature, co-related (switching option 09) Is ^ 
eiiri)edded In the synonyo option. However, In the_usj»p-^f1ned mode, 
option 09 can be selected Individually, hence the output can be 
labeled "corelated". 

(5) narrow stands for naMwer teras (NT). This label Is used to 
identify all the narrower terns associated with "your term". 
This option works In a limited siense in the DOE Thesaurus; 1t only 
produces narrower terms at the NT 1 level from the DOE Thesaurus. 
- This label corresponds to switching option 10 In the user-def1ne<d 
moie of YSS. 

(€) related ^tands for a related term. Related term Is equivalent to*^ 
the "RT". or related term relationship found In many thesauri.- This 
label Is used In conjunctlonj^mi switching option 21 In tlie user- 
mode of V$S. 

(7) rel v^msm stands for related phrase. Related phrase Involves a 
stmi1;ig prQ(:Qdure (right ^tnmcatl on only), the stemming. procedure . 
Is Applied ta^ery word' In the phrase enter by the user. Individual 
stems are combined (concatonated) Into a string and this string Is 
used to search a file of strings created In an Identical way from ^ 
all the VSS. vocabulary entr,1es. "Hits" from the st^ file then 
cause' retrieval of the full, unst^mied phrase frm another file. 

* 

Thus, the output phrase contains the stem or root of each word used 
In the input phrase. This label Is used In conjunction with switch- 
ing option 06 in the user-defined mode of YSS. 

(8) stm match stands for stem match. Stem match Is identical to word 
match except that stems of words are used Instead of complete words. 
This label is used in conjunction with switching option 08" in the 
user-defined term mode of VSS. 
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(9) sywony staiu^f for iytionyms. This type of tens Includes US . 
SEE, USED FOR, and SEEN fRON, cross references. It represents an 
.exiiaustlve 'look for sj^Qnjos in teth foriiard (USEf) and backward 
(UF} directions. It does not > Incorporate co-re1ataed^rms. 
(Notd difference between synonym tnd synonynH-). This label 
Is used in conjunction wjth switching option 19 In the user- 
; - defined «ode Df.VSS. J ^ 

(^^). syfionyir» stands for synwym plus. SjmonjuH' IncTudes USE, SEE, . 
USED FOR (Jf},, S^ from, "and co-related terms. In ut-her wortis, 
tt 1s an exhaustive look for s^onym in both the forward ^(USt) 
fnd backifard (UF) directions. See cdrrelated for a detailed 
description of the co-related terra relationship .1, ThU label' is - ' . 
used to -Identify output associated with "symmym'* and "browse", 
features tn VSS. This, label is not Used in the user-defined 
aiode of VSS. ^ 

(11) teras * conbi nes .two t^rs relationships used by VSS, "your 
tera" and "^ynonyi^*. Thus, a cooaon label, Is use< to designate twp 
types of term. This label Is used for all output associated with 
switching option 05 In the user-defined amie^of VSS. ^'Synonyn" 

In this case d^otes only the USE or SEE cross references foiind 
In various thesauri. This s/nonys desij^ation is anich more limited' 
tha?> either (09 or 10). 

(12) wd Batch stands for word match. In word natch, the search tens 
(or phrase) to be switched Is broken down Into Its component words 
and each word is i/sed to search a VSS n^rd file created in an 
Identical way from all the VSS vocabulary entries. Users mus^ 
specify how many words of the1.r original term must be matched in 
order for a switch to be considered successful. The default Is 100%. 
For ex^sgsle, if th^ search term contain^ three, vrards and two of 
the three words mist be matched, the us^er must set the percentage 
at 66. The system will retrieve all -phrases In the file that have 
at least two of the three words used 1n_ the original phrase. 
However, phrases retrieved by VSS may contain more total words than 
the original term or phrase entered by thd user- In other words, 
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the percentfige specified by the user Is applied to the Input 
phrase k not^ output phrase. Tills label 1;$ used In conjunc- 
tion, with switching option 07 In the user-defined node of VSS. 



(13) vmr tm Is self -explanatory. 
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E-1 
TESt SITE 2 



Subject ^^sduY^es for Conducting RetrlevaV ExpeHients 

1. Decide which query($) win becooe a p.art of the experlnent. Six queries will be 
evaluated duHng the month of Harch. . • 

(A) Deternlne If request Is^U Hatched with one of the VSS 
nodules. The aodules^^j^^: 

Buslnesi 

ABI Injurs 
Mg^* Contents 

* -* Behavioral 



Psyc.Abs. 
ERtC • . 



o ' 11 fet Sciences 

m • MeSH 

' NiSH* (Ch«i1cal Registry r^S.) 

Ne^" (Cheiilcal Substances)' 

CA (CEF) 
/ BIOSIS 

PhyslcaV Sciences 

DOE •. • • - 

CA - 

EL (COMPEIffiEX) 

INSPEC 

MASA 

(Trilingual) 

(B) Determine willingness of end user- to participate In the experiment. 
To participate, end users will be required to record their relevance 
.(teclslons at the time the search results are ejeamlned. {R « relevant 
or useful item; I - Irrelevant or j»bt useful Item*, ? « undecided). 
Battel le will pay for th« VSS search and half of the regular search. 
Tne end user will be required to tunuliBrfvaluated results for photo* 
copyjng to penrit subsequent atia lysis.. 



. *Not in VSS but dptlonal for searching purposes. 
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, TEST SITE 2 (Cwitlnued) 

If 1(A) and (B) are favorable, start the experiment at the search negotiation 
(pr^-search interviews) stage by bringing in the Second intermediary. This 
interview wilLconMst of an end user aiyi the twoMarch ^termed arles. Once 
theMnterview is complete end usgr trrtervention 5r interaction ^^^^^^ Pe""]>i;°. ; 
only for.tfie purpose of reviewing VSS output or the searth terms selected wi.thout 
the use of VSS.' End users interaction during the onlinf session will not Sfe per- 
mitted-. ' 

At ,Jthe conclusion of the intervlewv each Intermediary must pVt)ceed independently 
until the searches arp con^l^eted, » # 

A. .Ohe searcher. will proceed without VS§", the other withp'vSS. SOP will be employed 
* , -for all searches don^ without the* aid of VSS.., This may or may not Involve the use 
;, -,-Df printed thesauH.' It is ii!9>ortant that the regular Search be dofte as near to ■ 
SQP as. possl^Jle. , ^ . ' . . ' 

' 5^ -Both Intermediaries may Intervene w1$h their online searches to "fine tune^^hem* 
but 00 1^ if this is warranted and is done as , SOP.. ^Avoid thp teii|)tatibn to "fifte 
tyne" 4ust to made the experimental results better. " ^ ^ 



6. 



lach searcher will recordi the actual t^ime/requ^red to i)rep^re the record- strategy . 
• prior to Ibgdnv .Also, connents ma/ Bevjrecorded.both dupirtg' and after online search . 
.s'ession.' ^ ... • . " * , *» 

" ■ V • : ^ , ' •- • ■ ■ ■ . . ■ 

7. 'The searcher 'who does* a f»on-VSS- type search, w^ll print all results for evaluation • 
by- end. ustfr and store the' search profile; m ' : • » . 

8/*'The searcher.who do^ i VSS-tyi>e search will store the j)nof11e^fore. printing any 
. % results. Thefir in^rder to eliminate duplicates-, the non-VSS search^ profile will 
be subtracted (Boolean NOT) from the ySS-seerch profile. The results will be 
printed and sent to the ^nd user for evaluation. ^>s6, the intersection between . . » ^ 
VSS and nOri-VSS (Bobleen AND) wIlT be Iderttifled by.rthe VSS searcher, the accession 
numbers printed out^^ and t|^ results properly ictentl^led. Theie Acc. Nois. represef^t . 
the ^itatiops* tomijon to both stra'tegies but -evaluated on-ly once by the end user: 
NOTE: This procedure will only eliminate duplicates within each data base searched. 
It will not elJflrinate duplicate across data bases,. A sort, prior to printing-out . 
xJftta tasie titations is desirable for the purpose of eventually eliminating duplicates ' 
across data bases. " ' , * / • . 

/ ■ . . . ' • ' • * 

•9. Labe'T sU printouts according to the following ichei^: 



• * 
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1 : 


• 




['^ • VSS 1 


i 

mm 








VSS > 






: S«'Ch t 3 ; WS 1 

■ i • 

j . 1 fSS, , 


" 1 
rss 


f 






• 




'i §- • • 

— =^ 









In this scheme, each intermediary is" identified, by a letter code. A; or B.^aqd-' 

he/she keeps that .code throughout -the experiment. Each-search is nunjbered Sequeptlaliy 
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/ TEST SITE 2 (Continued) 



and each data base clearly Identified. Thus,. one end user" request, can^Jletely ; 
processed through the experimental aethodology, wight be >abeled as foVl«*s: 

\ ^ * Search VSS Searches. . . , 

\\^AE9iC . l-BERIC ;; ' '* 

' • V-A Psyc. Abs. 1-B Psyc. Abs. 

• • • 

. The end user will eyalOate all four searches for relevance and will use the notation 
descHbed In (l.B). This Is an liaportant detail. We don't want to interpret- 6^ 
different notation schemes for <tenoting relevance, non-relevance and undeclded-^ype 

decisions. * ' ' . > 

. " . ■ ■ f . • 

10. Have the end user c<w|)lete the brief questionnaire and fetum it along with the 
evaluated results, . . * . 

n . Return to Battel le: , 

(A) Intennediary's wof^heet with pre-search preparation time and coipents 

(B) Copy of |nli'ne terminal session 

(C) The list of accession niaabers coBpon to both VSS and non-VSS searches 
.■ . ' • (properly labeled) 



{d) The end user's evaluated search results 

* * ■ 

(E) The end user's Cof^leted questionnaire. 



RTN/tln 
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BUSIJ<ES$ • 

B ABI INFORM 

e Nanagedent Contents ^ 

SOCIAL SCIENCE ' 

D ERIC ' ^ 

P m (Psych. Abstracts) 



E*4 \ - 
TEST SITE 2 (jContlnu^) f , 

VSS Vocabulaify Ctxtes 

. PHYSICAL SCIENCE 

C Chemical Abstracts 

E EI r 

F Ifon (Trilingual) 

I iiePEC 

• N NASA . 



Suggested Strategies 





20 . 
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• 4 
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. 10 
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8 ■ 


7 
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7 
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STORE 


20 


STORE' 




. STORE 




EXPAND 




SCbPE 


•say? 




, SETPCT 


VOCCNT 


SETPCT 


50 


4 . 


66- 


VOCCNT 




VOCCNT 


.4 




4 



20 Expanded synonyiss 

6 Phrase file switching 
10 NarroMer terns 

7 Hard file 

8 St«R file 
12 Adjacency 

3 Exact aatch 

4 Linlted synonym 
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TEST SITE 2 (Continued)^ 
SEARDiER'S kmKSHEET 



Search Nwaber: 

^arch Type: a VSS ^non-VSS 
?re-search Prep. Ttne: w in. 

Search NunWr: v ' ^ 
Search Type: □ VSS □ non-VSS 
Pre*searph Prep. Tiae: w in. 



Search NuMber: 
' Sea rph Type: Q 
Pre- search Prep. 




a** non-VSS 
sin. 



Search Kiwber: 

Search Type: OVSS □ nonrysS 

Pre-search Prep . Time : roln . 

Search Number: 

Search Type: D VSS, b nqp-VSS 

Pre-search Prep. Time: win. 



Search Ni«i)er: _ 
Search Type : OVSS 



□ non-VSS 
Pre-search Prep. Tine: w in. 



Discounting the separate logon and switching strategy set-up procedures, nihlch 
^(telttedly are tedious In this experimental syst«a»,d1d VSS help reduce the 
mntal burden of search strategy preparation? ^, 



Uhat did you like about VSS7 



did you dislike about VSS? 



I 
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-y^ - S. 1-6' ^ f ■ ■ 



•t 



••• 

•• . V*, 1 . ■ 



USER'S EVAI.UATIOW SHEg 



(V) N.f'oip scar^'-'T v. . 'lid'.-you feel that you f»ceiyed about the right 
tewrit of' WWallt" teful G(itat1ons with your Initial 



titatlons : 



1 



' \ • ; ^ . e)^tat:tfl«5f?C^ Pte ' '■ „. 

vM. W«it, iipas'y^ ov^lt cl|i^ree^ of s^tt^^ with this search? 
. v: . Circle one.'^'^lf.. -^-^V ^yv".':.-:. . • ■;, .. 

• ■ • ■ ^ 2''^'"^'^ '- '^■^ ^'.^^^ ^ 

> - yary :V:,f Highly 

■ ; r^^pissilltlsf 1«d| SatHfled Satisfied 

(4^) f or'^aarch y , were tee addl ttwi relevalit citations ; 



it3 Indll^ V tvotfr th^ a^ltlohal. co^ and effort 



m » 



13^ 1^f4i1r 1.e/;:hlopy to;ha^ lKf.t could have gotten by without 



^ □ fipt ^o6,u^ni1, i.(B. , ot, ^rglnal or llttU value and probably 
. ^ ' 'Mt wbrth the coi,t>ar^'effbn to 1^ t 

{^) For the^saie sekr^hfwa^ the «bunt of Irrelevant c1ata*t1bns: 
*. O: ^oler4.ble □ Annoying q Didn't matter 
in*3|pur opthlont 

« . 

(6) What was your overall degree of satisfaction with this search? 
Circle one. 



TP" f.. 



1 2 3 4 

Very Moderately Mocterately Hery 
Dissatisfied Dissatisfied Satisfied Satisfied x 
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TEST SITE 3 * 
tSS E«geH«Mt«1 HBtfcwtelofly . 



DeterelfM If request Is appllcalbe to this experfjwnt: 

(i) The request «ist Involve YSS databases: 

Social Sclertce Business 

. ERIC \ ABI Inform 

Psy* Abs. Mgst. Contents 

Life Science Physical Science 

.BIOSIS eA 

BIOSIS-C DOE 

CA ' EI 

NeSH INSPEC 

,NeSH-R im 

NbSH^'S NASA 

(b) Request must Involve a search of tMO or oore databases*. 

Detemlne end user's ir1111n«tess to participate In the experiment. 
To parti c1 pate i' end users in 11 be required to (a*) negotiate their 
search request 1a the -presence of tuo sev^hers (Intermediaries) » 
(b) revlelf search tenss and strategies prior to the online 
session, as rraeded, (c) be agreeable to fo'rfelt .any Interaction > 

turing the' online search Itself, (d) evaluate printedi' citations 
or relevance (a nixInifiB bf citatlmts irtll be evaluated), and 
(e) co^ile'te a specially deslgiyd ^i^stlonnalre.' 

# 

Botii tntenaedlarfes conduct a joint pre-search Interview irlth the 
end user. All three jttople asfst be present at the start (end 
user and both searchers), ftecord search ne^tlatlon tine on 
Intemedlary Mrksheet. 

Both Intemedlarles. prepare search strategies based on pre-«earch 

Interview; one liitermedlatey win use VSS, the other win not. 

lyMtnWT ; The Intermediary who tioes n6t use VSS must not go out . 

of h1s/>»r way to-do anything extraordinary to try to look good or 

beat the VSS systen. In fact, there najf be tines i#ien the 

1 ntem^l a ry would not consult a printed tfmsaurus under certain 

circunstances. If this Is th^jcase, then a thesaurus should not 

be consulted for this experliei^t. This Is an honor system. 

DO iOr 06 AITTHliG OUT OF IHE /teDHMilT FOR THE SEAICH If you are 

the Intermediary who Is. performing the nc^i^VSS search. 
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TEST SITE 3 (ContlniMd) 

Step 5 End u^r "fine tiding" of the search will be permitted only to the 
extent of saying "Yea" or "Hay" to terms produced by VSS or non-VSS 
methods. Under no circumstances may a terra- used In one strategy be 
added to the other If found missing th|9re. Ikmei^er. In evaluating 
each term in each strategy, the end user may., in effect delete 
identical ^ terms froB each strategy. This is permissible. 

Step 6 the searcher using VSS raist do the following: (a) select all the 
vocabularies in the indule chosen, (b) select at' least eight 
terms to be displayed per vocabulary, and (c) perform switching on 
two VSS options, browse and nacrow/^iroader. - 

■ * Step 7 Intermediary using VSS must collate VSS Evaluation form for each ' 

VSS search performed. » 

\ ^/ 

Step 8 Intermediaries must record time for using VSS or manual- thesaurus 

and sfearch preparation time on Intermediary worksheet. I 

Step 9 . Wien both search strategies are ready for executitm as online 
searches, both strategies will be entered in their entirety and 
stored as profiles. Each strategy will be executed against each 
database requested by the emi user. Although duplicate citations 
canfiot be eliminated between da^bases, within the same database 
the following procedure must be followed: 

(a) Subtract (boolean MOT) the non-VSS profile-froo the VSS profile 
to produce Set A. Label the offline printout as SET A. Per^ 
form the reverse subtraction to produce Set C. Ubel this . 
printout as SET C. In otter words, SETA will always be clta- 
tims uniqiM to VSS and SET C will always- be oitations unique 
to non-VSS. . 

(b) If either SET A or SET C contains mon than 50 citations, 
limit both sets by date to reduce the nt»ber of citatlms to 
50 or less in each set before* invoking offline print. 

(c) Intersect (boolean AND) SETS A anc C to produce Set^ B. Ubel 
printout as SET B. Set B represents citations, common to VSS 
and non-VSS profiles. 

« 

Step 10 Present each set to the end user without reference to VSS or non- 
' VSS and without reference to the searcher's naan. Set B must 

' always be the intersection of both profiles. Set A imist always 
be VSS unique and Set C must always be non-VSS unique. 
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TEST SITE 3 (Cofiltlnued) 



Step 11 Have the end i»er evaluate each altatlon In each set (A, B, C) 
— ■ — for relevance to his rtquest. His/her decision* should be narked 
In the coluBRi Off the printouts using Rfor relevant, I for ' 
Irrelevant, and I for .undecided. RO ailKR MOTATICN SHOULD BE USED. 

Step 12 - Upon coapletlon of relevance judgeaents, end user oust complete 
— — . end user»s worksheet for that search. End user mist return 

f»rked-up printouts aAd worksheet to searcher fon photocopying.^ 

Step 13 . Return all worksheets, printouts and copies of online sessions to 
\ Battel 1 e f or anal ysl s . 

AgreeiBent Intenaedlarles should have slwllar capabilities, educational 

— background and online experience. Intenaedlarles agree to perfom 

a nlnlRUB of six searches over a 6-week period In dual or replicated 
fashion, as follows: 

Ifflkfedlary 1 IwtenMdlanr 2 

Search 1 VSS^ . ^ Non-VSS 

. ' Search 2 Non-VSS . VSS 

Search 3 VSS • Mon-VSS 

Search 4 ' > Hon-VSS ^ VSS 

Search 5 VSS Mon-VSS 

Search 6 * Hon- JSS VSS 



i 

4* 
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(1) 

(H) 
(3) 

(4) 



E-10 

TEST SITE 3 (Continued) ^ 
XNTEMOIARY'S MORKSHEET 



Institution I.D.: 
Intenaeidiary I. p.: 
Search I. D.: 



VSS Module used: 

□ business 

□ social sciences 



O life sciences 
□ physical sciences 



(5) 



PERFORMANCE DATA FOR THIS SEARCH 



Pre-Search 
Preparation 
(In iM^fiutes) 



Online 
Connect Time 
(in minutes) 



VSS 



xxxxxx 



Iton-VSS 

Search 
^tegotUtlon 

) 

4fSS Online xxxxxxx 



Thesaurus 
Look up 

Strategy 
Preparation 

' TOTAL 



VSS 



Non-VSS 



I» 1 
2 
3 



TOTAL ^ 



(6) CoBinents about this search. Including any difficulties encountered 
(VSS or Non USS): ' 
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TEST SITE 3 (Continued) 
END \&WS UCmiCSHEET 



(1) Institution I.D.: . ' 

(2) End user I.D.: 

(3) Search I. D. ^ ; 

(4) . State your search request in your om words after you have discussed 

It with^ the person who will do It. 



(5) In your past experience with online searches, are you generally: 

□ satisfied with the results 
f * 

□ dissatisfied with the results 
D' var^s from search to search 

□ no past experijsnce , . 



(6) , ^ • . PERFOWWICE DATA FQlfr THIS SEARCH 



Group 


NtARber of 
Relevant 
Citations 
in Group 


Hun^/of 

CiSrralnsr: 
in §mip 


Number of 
Citations 
Undecided About 
In Group 


Total 
Citations 
in Group 


A 










B 


• 


• 






C 






* 





« t 
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(7) 



(8) 



TEST SITE 3 (Continued) 

Q«0 USER'S WmKSHEET ' 
(Continued). 

^ . i ■ . ' 

Rank tfie foTlomng cltatlo^i groupings according to your overall 
degree of satisfaction: 



Citation Groupings ' 
Sroup A plus Group B 
Group B plus Group C 
G^oup A plus B plus'C 

1 » very disdptlsfled 



Satisfaction Scale 
1 2 3 4 5 
1 *^ 3 4 
12 3 4 
5 " very satisfied 



If you could select only one conbl nation shoMn beloM . 
which coH^lnatTon would It be? 



Group Coirt>1 nation ^ 

Group A plus Group 6 ' 
Group B plus Group C 

Why did you choose this carina 



■ 

Choice (check only one) 



(9) If you had to pay extra for the combination you selected In 
Question 8 ct^red to. the combination you did not select, how 
Biuch extra would you te wining to pay? ^ 



$5 



$10 $15 



$1S 
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f 
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F-1 

VSS EVALUATION FORM 



\ 



tl) Institution I.D.: 
(2) l|ser nam I.O.: 



: — : — : : 

(3) Your Institution can best be described by oiie of the foil owing: 

Private sedtor \ ■ . a 

O . independent broker * ' 

□ corporate llbrary/inforwation center 

O non-profit institute library/infonsatlon center * 

□ database producer (primarily) 

□ online search service (primarily) « 

□ database producer/online search service (about equally) 

6overf«»nt (Federal) 

* . • • ■ i" •• 

□ librafy/lnfbrmation service 

□ -database producer (primarily) » - 

□ online vendor (primarily) 

a database producer/Online vendor (about equally) 

Acad«Bia 

□ library/ information service 

O library/information school or department 

Other ^ 

□ public library/information service f.' 

□ ; — 

(4) Your Academic background: Field of Study 

Degree i for Each Degree 

Kto Degrefj^ • - \ 

Bachelor's > ■ 

Master's . ' 

* Doctorate • 

Post Doctorate • 



(5) Years at present company:. - 

(6) Years in library/information activity: 

(7) Years experience with online systems: 
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VSS EVALUATION FORM 
(Continued) 



On a scale of 1 to 5. rank the databases listed below according to 
yoyr current proficiency with and usage of then. Please rank all 
choices listed. If you currMtly'*do not use a particular database, 
so Indicate this, ^nd 1gnor«he proficiency scale for that one^ 



Data Bases 



Your Proficiency 



Frequency of Use 



ABI Infonn 


1 


2 


3 


4 


5 


1 2 


3 


4 


5 




1 


2 




4 










c 


ERIC 


1 

1 


2 




A 


i 


1 2 


3 


*f 


e 

w 


Psychological Abstracts 


1 


2 


3 




5 ' 


1 2 


.3 


4 


5 


imiNE 


1 


2 


3 


4 


5 • 


1 2 


3 


4 


5 


6I0SIS 


1 


2 


3 


4 


5 


1 2 


3 


4 


5 


Chemical Abstracts 




f 










/ . 






(CA Search) 


1 


2 


3 


4 


& 


1 2 


3 


4 


5 


Compendex 


'1 


2 


3 


4 


5' ' ■ 


I 2 


3 


4 


5 


INS|EC 


1 


2 


3 


4 


5 


I 2 


3 . 


4 ' 


5 


NASA Recon 


1 


2 


3 


4 


5 


1 2 


3 


4 


5 


,00E Recon 


1 


2 


3 


4 


5 


1 2" 


3 


4 


S 



1 - little or no proficiency 

2 " below average proficiency 

3 ■ average proflclehcy 

4 * above average proficiency 
& ■ expert 



1 > no usage 

2 » less than once/im>nth 

3 « one or nore tln^s/sionth 

4 " one or sore tiaes/weefc 

5 - dally usage 



How often do you search with the foil owl ng^ 
(answer should tot^ 1002} 

a. controlled Vocabulary terms only * ^ 

b. controlled plus- free text terns ^ 

c. free text terms only ^.^ ^ ^ 

Indicate your preference below (P.lease rank "I" fof most preferrwi, 
•*2'* .for next "most preferred, and f3" for least preferred). : . 



% 



% 



a. 
b. 




vocabulary '4iems only 
us* free text tern? 



^lly your, actual usage of VSS during tl» test period. ' •» 

Search No.: 1 « , Datd: 

Search statement: - v 

• circle VSS vocaMaHes selected for th^ls search: . 
(refer to VSS online nenu) , v , 

(1) A B . (2) A .0 (3) A B C B-E F (4) .A . B C D E - 

• Circle VSS switching features selected for, this -search: ^ 
(refer to YSS onl ine iienii) . , • / 

• In :Z ; .3 - 4 - • 5- .. . 6 . 

• Mere yoti'ahle to use ah/ ^ the VSS results for this search? *□ Yes □ No 

• -How sany^VSS search tetitt or concepts were you able to use?, ' ». 

e ' was VSS helpTul in suggesting any other □^'^•fiiis or □ avenues of approactTTbr 

li^rovlng this search* beyond what the systea actually produced? Q Yes d No 
e In. your best Judj^eiiit, 1^ wovld VSS output coiapare to your own noraal effort for 
fills swTrch? " . : ' • * 

□ VSS output would probably not-bO 'es good as ^y own nQMl effort 
d VSS output would probably be better than tuf o«h nomav effort 
^ a VSS output" would i)rol£»bly <be about the sas^ as py om noraal effort 

• How would -rate VSS for. this 'particular search? * 
1 2 3 ,4 5 (1 ■ of no hel|r ' 5- very helpful) .J- . . * 

• Coiments: . ' ^ V 



^ * 



■ \- ■• .... 

Search :No«.: 2 , ^ , Date: 

$ear'ch Statementr * V *' • . 

•Circle VSS vpcabularles selected for this search^ 

(refer td ViS'onllne wend) . - i J ^ . ' , 

(1) A. B \ (2) A,- B (3) A -B C D E F% ' (4) A B C ^D E F 
Circle VSS switching features selected for' this se#%h: 




(refer to VSS online raemi) ■ 
1 2 3 4 . 5-^4 - " V V 

• Were you we to use any of the VSS results for thi|.s^il*ch? □ Yes □ Nd ' 

• How inanylfSs search^ terws or concepts were you'^able to uise f ' 

• Was JCS helpful In suggesting >any other □ terms or o avenues of approach for 
ij^rovlng ih1s"search» Jseyond^Mliat the 'system actually produced? c Yes Q No 

• In your best Judgement, how would VSS output c(^are-to your own rfonaal eff()rt for 
' this search? * ' - • 

VS^ output ^ul,d probal'ly hot tie, as good- as own normal effort - . - 

li VSS output would ^r^bly be better than i^y own normal effort 
0 VSS output would prob^bLly* be about the sjkm as^ own^ normal effort - 

• How would you rate VSS fo> this particular search? ^ * 
- '1 2 ^ .A^ 5 . (l|S|f no help 5. - very helpful ) 
»*, ConBsents; v . ' zT^ . \ *; 



(11) Tally /our actual usage* of VSS during the test period. 

Search No.: 3 v v ^ Date: 

Search Statenent:^ 

'.. • ^ i. ' . .... 

' e , circle VSS vocabularies selected for this search: . ^ 
' (refer to VSS oifllne aefiu) 

. (I) A B (2) A B (3) A B ^ D E F (4) A B C* D E F 
e Clfrcle VSS suHtchlng features selected for this seirch: 
(refer to VSS oiillnfr aenu) 

1 2 3 ■ 4 5 • ^' ■ • ■ ^ ■ — . • • 

I Were yoii able to use any of the VSS result for this search? □ Yes □ No 
• How wany VSS search teras or tmcepts »iere you able to use? 
' e Mas VSS helpful in suggestlne any other tones or □ avenues of approach for 

Improving this search, be)imi what the .systen* actually produced? □ Yes □ No 
e In your best JudgMnt* hfm would VSS .output cos^re to your own nomal effort for 

this searchf 

□ VSS output Mould probably not be as good i$ my cm nomal effort 
' □ VSS oiJtput would probably be better than ay own normal effort * 
O VSS output would probably be alK»ut the same as my own norm 1 effort . 
. ' a How iift>uld you rate VSS for this particular search? » 

1 2 3 4 ^5 (1 - of no hel|r -5 ■ very helpful) 
- - • Conents; * 



0 « 



•Search No.: 4 Date: 
Search Stateiaent: 



\ 



• Circle VSS vocabularies. selected for this search: 
(refer to VSS online aenu) 

V (f) A B ^ (2) A B (3) A B C 0 E r (4) A B C D E F 
t Circle- VSS switching features selected for this search: > 
(refer to VSS online fc|u) - / ? 

*1 . ^ 3 4 5W 6 

• Were you able to use any of the VSS results for this search? a Yes a No 
• " How many VSS search terms oY ccHicepts were you able to u$e?_ ^ 

• was VSS helpful In suggesting any other □ Uptm or □ avenues of approach for 
. Improving this search, beyond what the systea actually produced? *n Yes CT No. • 
t In your best judgmnt, }mw would VSS output con^are to your own normal effort for 

this search? , * . 

O VSS output would probably not be as gbod is ay own normal effort 
□ VSS output would "probably bfi better than ay own normal effort 

* □ VSS output would probably be about the SMie as ay own normal effort 

• How would yOU'Vate VSS for this particular search? 

. 1 2 3 4.5 (1 • df ^0 h^lp 5 - very helpful) 

• Comments: . . ' 



F-5 



- . .' ^ 

Tally your actual usage of VSS during the test period. . ^ 

Search No. 1 5 . • Datef \\ . • 

Search Statement: ' • 

* ■'• * « . 

e Circle* VSS vocabularies selected for this search: ** . " 

(refer to VSS, online mnu) , 
^ (1) A B (2) A B (3h A B C D E F . <4) A B C D .E . F 
e Circle VSS switching features selected for this sell rch: 
(refer to VSS online nenu) * 
1 2 3 4 5 6^ 

• Mere ^u able to use any of the VSS results for this search? Q Yes o No 

t How »any VSS search teriis or concepts were you able to use? •• . '-- . t 

• ' Was VSS helpful In sugi^stlng any other □ tenas qi^ □ avenu^^ of apfmwch for ^ 

is^rovlng this search, beyond what the systes actually produced? □ Yes □ N6 * 

• In your best judgeiBBntt how would VSS. output coiapare to your pwn* nomal effort for 
this search? ' . ^ ; 

a VSS output would. probably not be as good as i^y own normal effort 

□ ,yss output would probably' be better than oim nonaal effort ^ 

□ VSS output would probably be ^bc^t the sas» as own nohaal effort ^ 
e How would you rate VSS for this particular search? 

1 2 3 4 5 (1 • of no .help- 5 - very helpful) * 
e Coaaaents: • . . * 



7 



Search No.: & Oate: * ^ • 

Sear.ch*Staterwnt: 

9 Circle VSS vocabularies selected for this search: 
(refer to VSS- online menu) 

(1) A B - (2) A B - (3) ABCDEF (4)ABCDEy 
e Circle VSS-swItchIng features selected tor this search: 
(refer to VSS online menu) • 

1 2 3 4 5 .|6 : . ' . " - * 

t 'Were you at)le to use any of thj^ VSS results for this search? * a Yes a No ^ 
e How Biany VSS search terms, or concepts were you able to use ? r. 

Was VSS helpful In suggesting any other o terms or* a avenues of approach fdr 

Improving this search, beyond what* the system actually produced? o Yes □ No 
e In. your best judgdment^ how would VSS output coi^>are to your own , normal effort for 

this search? ^ ^ , ^ 

□ VSS output, would probably not good as my own normal effort 

VSS output would ppoSably be bettiir than my own normal effort 

"a VSS output would probably.be tfbout the same as own^normal effort . 
4 How would you. rate VSS. for this particular searchf 

1 2 ^3 : • 4 5 (!' • of no help 5 • very helpful) 
e Cosnents: ... 
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(11) Tally your actual usagt of VSS during the test i^Hod. 



Search No.: 7 
Search Statesient: 



Date; 



A a C D £ F 



□ No 



Circle VSS vocabularies, selected fir this search: 

(refer to VSS.onflM aenu) % 
1) A B (2) A r (3) A B C 0 E f (4) 
Circle VSS sirltching features se1ecte<f for this search: 
(refer to VSS online aenu) 
1 2 5 4 5 6 , 

Were you able to use any'of the VSS results fqr th^ searxb?^ □ Yes 
How<«any VSS search tersis or cwicepts wre you able to use ? 
Mas VSS helpful In suggesting aiiy other ' □ terns or □ ayeniras of a|)proach for 
1iq»rov1ng this search, beyond ^t ^ systea actually produced? a Yes 0 No 
In your best judj^isent, fimt would VSS output coe^re to your mm nomal effort for 
this sjearch? , . % • . ^ ' 

Q VSS .output «(OuTd probably not te as food as ky cm ndrpal effort 
' 0 VSS output would probably be better than my om npraal effort 

□ VSS» o^tlput would probably be about the same as aiy mi nonftal effort* 
l)ow would you rate VSS for this particular search? * 
1 2 3 4 5 (1 * of no telp . Jji^ery helpful) 
Conaents: " ' 



Search No.: 8 
Search Stateaent: 



Date: 



B 



C D E F 



□ ^es □ No 



e Circle VSS vocabularies selected for this search: 
: (refer to VSSvJbnllne lenu) . 
(l)t A B ; :{2) A /B (3) A B C D E F (4) 
ii Circje VSS switching f$ature$ selected for .this search: 
• (refer, to VSS online awu) • * 
1 2 1 V ♦ • 5 • ; 6 
e Here you abU to use any Hf the IVSS resiclts for this search? 

# How ttahy VSS -.search tjiras pr c«ficept» ipiere you able to use? 

• Mas VSS helpful In suggesting any oth^r ^ a tenw or □ aveni^s of approach for 
• .linprovlng' this'searclsv beyond what the syst^ actually produced? □ Yes □ No 

e In your bf^t judgeaientf how wpuld l|3 putput coi^iare to your'oiei normal effort for 
this search? . ' * , - 

O vSS output ^wouMl^robably not be as 'good as ay own noroal effort 
□ VSS output wDulil pi^|l!y'::be better than my cm norrnl effort 
.0 VSS output ^lutd (irpbabljf be about the saaae as Biy own nonaal effort 
e ^ would you rate VsS fer^N this particular search? " 

1 2 3 .> 4 5 ^Jx* of no help 5 • very helpful) 
e iosaients: j^ / ; i , \ v , 



Tally four actual usage of VSS during the test- period. 
Search No.: 9 ^ . .Date: 

Search Statement: ^ ^ 

* . ' 

e Circle VS3 vocabularies selected for this search: ' 
(refer to VSS online nenu) , - 

(1) A B (2) A B (3) A B C D E F (4) A B C D E F 

e Circle VSS switching features selected for this search: 

(refer to VSS online «enu) * • ; 

I 2 3 4 5 « . % 

9 Were you>lrle to use any of the TSS results fot* this search? □ Yes o fifo 

• Howjiany VSS search tenas or ccm^i^ts i«re'you able to use? , 

• Was VSS Iwlpful In suggesting any otiHir □ ter»s or , □ avenues of approach for 
. iiqyroving this search, beyond what the system actually produced? a Yes □ No 

• In your best JudgMnt, how would VSS output ^di^re to your own nonaal effort for 
this search? * 

a VSS output would probably not be as good 9$ own nomal effort 
□ VSS output would probably be better than own noraal effort ^ 
• a VSS output irauld probably be about the sams as my own norsial effort 
e How would you. rate "YSS for this partlc^ilar SMrch? . 

1 2. »3 4— 5 (1 «| of no help, 5 • very helpful) ^ 
e • Connents: ' . ! ^, . ' ' * ' 



Search No..: 10 

Search Statement: . 



e Circle VSS vocabularies selected ^or thisr search: 

(refer to VSS online menu) ^ 

(1) ,A B (2) A B (3) A B C D E F (4) A B C D I F 

• Circle VSS switching features selected for this seai*ch: : 
(refer to VSS opllne senu) 

1 -2 '3- 4-56 ■ , 

9 Were you able to use any of. t^»e VSS results for tMs search?' o Yes □ No * 

• How many VSS search terms or concepts were you able to use? . 

t Was -VSS helpful In' suggesting any ottwr a terms or m3 avenues of. approach for 
la^rovlhg this search, beyond what the^ system actually produced? □ Yes .a.No 

t In your best Judgement, how would VSS output cOs^are to your Own norma] effort for 
thts search? . ' 

' a VSS output would probably not be as good a^^ own normal effo-'t 

□ VSS output would probably be belter than my own normal effort. 

□ VSS output would probably be. abMt the same as my own normal effort 
e How would you rate VSS for this i>flrt1cular search? 

1 2 3 4 5. 1 (1/ of no help § • very helpful) 



Coesents: 
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F-8 

YSS EVALUATION" FORM * 
' ■' (Continued) 



(IZ) Was VSS easy to: 
(a) learn 
'(b) use 
U) understand 



1.. . 


2 


3 


4 S 


1 


2 


3 


4 5 


1 : 


2 


3 


4 5 


- ^ease 






5 ■ hard 



03) Oo you feeV confident wtth VSS's: < 

(a) capabilities . X ^ * 3 4 5 

(b) output 1 '2 3 4 5 - • . . 

1 • confident <— » ^ 5 ■ i|o confidence 

(14) I,f YSS were available as another feature of present online search 
services, Muld It nake your job' of searching 'multiple databases: 

O easier n harder " o no difference 

Conpared to your present |Mthod(s) of online searc)t1rrg. 

Why? •'■ * 



^ (15) WhAt was your overall reeactlon to YSS? 



(15) What da- you think about subject SitflVchIng In general? 

I • . . . . . 

I A 



(17) tc 



tmmntsi 
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14. IF vss MERE n^MUms lis 'fmtm mtm W msm mim ^smcH 

SERVICES. 1IQUL0 rr NMCE rm m (T SEAfl(»^^ 
EASIER mSBt NO DIFFEREKCE 



Easier. There would no ^<m^r be a need to look through several data base 
thesauri. Pfy search area would be neater.nnd thus I could^lnk clearer 1 

Easier. I wmild not need to ke^ paper thesauri for every datable I use. 

No difference. Ywir systew doesn't really saffOs: t^"* experienced 
searcher who knows his or her thesauri. V A 

No difference. I use controlled «ter^ and free text now. . ■ 

. I- . • , • . ■ . ./ . ■ . ' 

Easier. U would sake my Job easier only If seldoM used, but expeffSlv^ 
thesauri were a part of VSS, andVlf It were Inexpensive Itself, ffy Job would 

be made easier 1h that I would have to' Justify fewer expenditures. \ 

... . • - •- • .. . . ^' ' • . * 

Ea&1e«e. It «iables -iMi to deterslne the entries to search mt -anotter data base 
If I cannot get the Info I need <m the present data base. 

■ - ■ j •■ ■- . 

Easier. Faster. UoMld not have, ta loofc up preferred terms. 

Easier. No CoMsent. r s, • 

Mq ansirer. No consent. • • • 

No "illfference. With free text searching; the majority of hits cow* from 
-these terns, not so miph^the ccmtrolled vocabulary.: . 

No difference. I search In life sciences and need the NT/BT feature so VSS 
would not help and so I couldn't use It. ^ 

No dlfferef^e. No cawnt. : . 

Easier. Saves tine'. 

fko difference. No conneht. ^ * . * . 

Easier. It can't hUrt, only help. " ' 

No difference. 6RS's CROS and DIALOG'S File 411 have provided sufficient 
guidance to potential files ~ at a reasonable cost ami quickly. ERlC's print 
tffesaurus is used fre((uently before going- on-line. 

No ^fefence. No coMnt. * « 

- • • • ft 

Harder. Yet another protocol to learn and deal. with. 

Easier.- Print thesauri for databases are of ten not coBwwly available. 



S-2 . . 

it' 

Ko difference. It Is very time consuming, and relatively unresponsive even with 
terms one would expect to be 1r^ the- vocabulary. Some of the terms were Just 
plain garbage! 

Somewhat easier. Useful In areas In which I am aware of only one or two. terms 
and VSS Is able to come up with related terms that I was unaware of. 

Easier. Did suggest more terms ^ a couple of cases. 

No difference. Not clear enou^ how switching works. Too cumbersome to use In 
present form. ^ • 

Easier. I would not always need to refef* to thesaurus — could even get by 
wHhout having so many. It Is a quicker more efficient way of checking 
terminology across files than using hard copy thesauri. ' *^ 

No difference. I <ton*t think we'd use. It as is. We'd still have to go to 
.pr1<^ed thesauri for notations (esp. mesh) and we'd still prefer to pull up 
citations and look at descriptors. 

No difference. Approximately 65?] of my searching Is free text. Though VSS> 
would contribute some additional thesaurus terms on thsi whole — It's still 
free text searching that I require. ^ 

,No difference. May make 1t^ somewhat easler^ut I think I more or less Icnow the* 
terms I.need. If I didn't. It's easier to print out the descriptors -In' the 
data base I'm already, t/i. V ' 

Easier.. Assist In locating controller terms for file we don't yet have 
thesauri for. ^ . ' . " 

.No iilfference. Because I usually use both free t^t~ «and controlled 
vocabularies. - . 

No difference'. No com^nt. . i . , - 

* • • • 

No answer. Application of 'thesaurus .terms varies from database to database. 
' e.g.. EI assigns only a fefi brMd terms.' INSPEC Is moe>e In d^th. only 
when concept not retrleveable in Tl^or AB. To only search the re field often 
misses relevant hItSt Stuategy muVt vary f ^)m DS ^o 89. ' ' 

Easier. I can search several data ba^ at once; 1^ gives nfe a "lead* Into 
data bases I noffttsl 1 iar with. ^ * 

Easier. Not. all data leases have an online Thesaurus^ 

Harder. Present methods : (1) produce more relevant references. (2)*^nswiie 
less time in fonmjlatlng a search strata, and (3) Incur no addltlonp costs 
for online connect time. - . 

. • ■ • • • • •• -".» ■ - 

No difference. Text searching of title and abstrac^'^f or terms or ^phrases one 
•wants to- see In title or abstract •'will produce citations of Ir^ii^est. A 
display of these will provide Iniiexing terms and ildentlflers foe 'that data 
base. ^ ' . 

J ' 
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No difference. Th|;^^ of format used ,1s tiae constffliing* 

Easier. Extends the EXP/WTO capabilities by aerglng files. Does a very good Job 
of relating terms If the database producer has built a BT/NT thesaurus. 
Outside the scl /tech Tiles, VSS*s usefulness Is decreased. 

Harder. I'd still check manually. Your system doesn't really save time for an 
experienced searcher who knows hl&or her thesauri. 

Harder, until It was reloaded to exhibit all thesaural characteristics. 
BT-NT first In browse rework adj. and stem match. . 

Easier. Only If It were available for databases that lack good thesauri. 
Nearly all the databases Indexed have excellent thesauri, which I think are 
more easily used one at a time than combined. Al$o» the thesauri have certain 
features that are not Included In the VSS. If you put such a system together 
for databases with no thesarus, or very costly thesauri or nultlple thesauri 
(NTLS, for example), then It would be extremely useful. 

No difference. We have on-line search aids that basically serve our needs. 

Easier. Easier to locate terms In other v(N:abular1es. 

No difference. If there was no match of terms among thesauri, there's ho' place 
to go from there. If I'm going to rely on exact matches, why not just free 
text across d&tabases? 

f • • ■ • ■ ' • . 'r 

No answer. No comment. . 

No difference. I spend a good deal of time In the - Interview with the patron 
; examining thesauri, developing synonym lists ^and, plain figuring out back*^ard 
' variations to sneak up on topic. , • . * 

Easier. Because is brings together the terms u^ed by ditferent thesauri ,1 it 
helps in the selection of appropriate terms for, effective 'searches of 
particular databases, " " 

Harder. Introduces^ an additional "head" in tjj^ way. Findfng valid "hits" is a 
matter ef onderstanding the^ need — the database, the s(^rce of the database. 
Having one more "help" doesn't really 'help. . f 

Harder. Use of this system represents a substantial amount^ of time that could 
be better spent searching traditional printed sources when composirig an on-line 
search strategy. . » * ' . 

No dTf€erence. ^o ccmiments. . _ 

Harder^ There fs very little infonaation received by the user, which is 
valuable- for switching purposes. 

*No difference. I probably wouldn't use M y&ry much, as free-text searching 
works almost as well. « 

• . ' . ' •» 

Easier. Ky brief experience with ySS did provide more pos%?ble search terras. 
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No difference. Its data base Is too limited. 

No difference. I would not use It In databases for which the vocabulary was 
familiar to m, and I would need a far a»re detailed explanation of what to 
expect from the systen, and how to use It. 

Easier, If you don't predetermine subsets. I want to chose the databases to 
cross-search. 

No difference. I need sore Information on how It matches and retrieves terms. 

No difference. When only 2 or 3 databases are to be used in^cfolng the same 
search. It Is Just as easy to consult their printed thesauri. Moreover, the 
scope notes provided In the printed thesauri can give a better Idea of the 
appropriateness of using a particular term. 

No difference. I can already use' D1al1nd«( which not only covers controlled, 
but also free-text vocabulary and also gives me an Indication of the postings. 

Easier, kftien you are switching databases. It Is helpful to know In advance the 
differences In the terminology.. . . ^ 

Easier. Would not have to look up terms In thesauri. 

No difference. No comment. 

Easier. When switchfhg databases and v(K:abulary, It would take the guess i^oric 
and/or checking printed thesauri out of the searchli^ process.^ often >^,$w.itch 
from ERIC to Psychlnfo with "end/savetemp" and this VSS system could eliminate' 
the less than successful switch by giving the right vocabulary. Aisfi in search 
strategy, we could use the VSS as proi|»ts for alternative sear^^stri^ejjles^ 

Easier. Good selection of terms Is suggested. 



No difference. Most of the searching done Is on scientific data bases wlt^ 
which I am' thoroughly fmrfllar. the system would be of value «^en T have to 
search business or social science data bases. . > ^. , . 

* • » - 

No difference.* VSS gave me little, if any, additional 'fYiformatlon. 
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15. mT MS YOUR OVERAL REMmON 70 VSSt 



Even though use was llslted, I Mas Impressed with the VSS. The system Is 
relatively easy to learn, quick, and reliable, logging on Isn't easy. 

Since It Is only a subset of tents & neaj. coaaerclal VSS could be. It did not 
score highly In the Biological, area. Its strength must be In the.'Jiard 
sciences. * ' . ' 

Increasingly negative. 

Interesting. 

Interest. It appears to be a first step, fi^ the third or fourth generation. It 
should really be worthwhile. But it mst be very Inexpensive to use. Unless 
you the thesauri updated — wiffiiirt fall--VSS will make more enemies 
than friends. 

tan be useful If you access a nmb&r of data bases frequently for fllfferent 
topics. ^ ^ ; 

Vefy favorable. It aw^wed ^o worH as advertised. ^ 

« * » • • • ? . 

From what we have seen, VSS looks like it could be a valuable searching aid. A 
better evaluation trt^t be made If wfr kngew oore about how It woiild integrate 
with an active on-ltne systeo (e.g., DIALOG). 

No content. . ~ 

It was handy, "but the message to choose an 'optlon after each result— became 
tedious. It 'would be helpful If ywi could stack comnds to get over the 
repetitive message. . • V 

I doubt If It Is worth what It costs. 

Tiflditterent - 

Very positive. It seems as if It Is a straightforward . . relatively 
uncomplicated tin execution, that Is) system that "can aid searchers by sav^lng 
time and being comprehensive. * 

* . * ' * 

I didn't feel It was very useful for everything X needed it for. 

Some of the options didn't work — As In more Info available. 

Interested at first but the^.results from the: "quality circles",' search began a 
negative trend. , * 

No comment. * ■ . 

✓ '■»'■- ' ' . 

There were not enough s^onyms to be of real asslstih^ce; nor were there enough 
hierarchical relationships represented ta offer assistance. 
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There seemed to be some progrsBmlng difficulties and the liianual was not easy 
to use. either as a reference or a learning tool. 

I got very tired of the menu displays and the Yearns of paper output generated. 
???????? would be helpful; also it aiH»eared that you. had to logoff and back on 
again to reach a different file group — rather Inefficient. 

Interesting, but mildly disappointing. Perhaps greater familiarity with the 
.System would'" have led to better results, "but too many of the switches led. to 
nothing useful. < , 

Too user friendly — I got so bored waiting for it to list the few menus I 
could choose /from. Heedi an "expert" mode (6r "semi -expert"). 

Good Idea -f heeds work. \ • 

Positive - for reasons above. 

Negative - switching features part not necessary. Can't alj. terms be 
listed for a term at once (broader, narrower, synonyms, etc.) 
Instead of having to select qf>t1on each time? 

Seemed a good idea; primitive stage; seems overwhelming to contemplate how you 
cpuld ever make it really useful by entering any free text terras— have to have 
broad. descriptors' only somb preknowledge required. 

" . . J"' - ' ' . • • \ ■ . ' ' 

I liked it. If it was se^ up so that one rould save terms and then go onto 
thos^ .databases and run tlie search on thQse terms. 

Seems cumbersome. * * ; * * . / 

ySS would.be much more valuable with vocabularies having troader and narrower 
terms — I continued my searching to the Business Vocabularies. * 

Need more experienc^ As with alT systems, 1^ takes practice. 

'ito comment * . . • 

Should; be cont1ou«l as a. project. Have you considered using REPOIT OISPU\Y 
feature of BASIS, tp provlitte tabular display? 

I thought it was a good system; - 1 don't like the "synonym" feature, but 
"browse" was .excellent. ... 

There is a great (^al of information but it is poorly presented and out of 
.date. . . • ' . 

VSS is too time consiaiin^ for productive onlin^earcjiing. 

1' '* ^ ' 1 • 

Hy overall reactions to' VSS H that the end result is' not worth the effort and 

cost ^to reach higher correlation of imtexing terms between data bases. In a 
majoflty of instances the switch Is direct or can be by truncating. The added 

. step required by VSS .equates tp timi arfd cojst increa^ses far in excess of 

benefits. ♦ * 
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. VSS would be more useful If "the vocabularies iiere continuously updated. Many 
' * of the. current subject term are missing. * 

Bro**s1ng did nof help significantly while doing a subject search; wight- be 
'TOre useful with author's, co^any naner,. etc. Synonyms, broader/narrojter 
fer«5. gobd to the ^legree of the vocabularies. Input. , Opinion »€ needs to.be 
: explainer! wore <:o^letely Wore I CQj^ld* judgei ' , : ^. 

There is saw potential here. • 




Nocomndht. . . * ' ' ' : ^ 

Ue were not able to. use VSS to-its full capacity because of difficulty with ' 
. t>jwjet. We also had to bdh*ow a 300 baud ter»1n,al because, our terminal (TI 
OMII-820 KSR) cannot be set at half Implex. * '/ 

Instructions i«re difficult to understand. Consequently, I 9gn*'t think I was 
. abje to explore VSS c^abllities adequately. 

I found the nenu tiring. I'd like to be able to combine databases in a ^ 
different way (e.g., Psyc, MSK, ^and BIOSIS or Psyc, Mgt., ABI).- I'd IJke to^ 
Be. able to add' narrower -terfss to the browse. featiJre. I found* the "switch -neir. « 
successful" response unlnfwwatlve— why i^ras It unsuccessful? ^* * 

No comment. , ' „ * 

\V A .. * . ■ ■ ■ ■ . ' 

Not at all iapressefd. - • • * 

I was 4a|)ressed with its switching capability.. Matches' i«re fbund for' ^tm 
terms and most of th^.ifere relevant. It would be expeciallst uiefuJ for an end^ 
. user OP new searcher unfamtliar with the terminology used by particular data 
* bases.. .> ' • ° ' - • • . •'•..*■ 



Negative., y ' . - - . , ' 

I was' lBpr6s$ed i^lth ^he technical accomplishment that this system represents 
but npt with Itr practicality or utility In the on-line database searching, 
context. > ^ - * ' , • ^ - . : * " ^ 



J. 



Menu characteristics frustrating, flot particularly user friendly. , 

, *■ ' ■* ' *.'*•'. . • 

it is a beginning but more * developmgh,t of the idftware is mandatory if it \t 

to function in a reference capacity; ' ^ \ • v 

It was not worth the effort, aTttod§frTt--wdiUd ^improve over as it became 

easerHo' iKe« j: , . * • / . - , ' ■ 

^ •■ -« , . y. » - ' . *» ' _ , > ' • ^ ■ 

I woold 1 ike to .try . 1 * 

Very limited value. . / , - / f^/- 

Very frustrated--t»i« repetition of tlie mMu Is fri|trating-«|nd tiS'Conswnllia 
^d theue are too aiany tferm types te ke^ stral^t ^Ith an .initial use. 
Feature 6 might -be mor^ us^uT «1f the com^nds and optlons.were explained more 
fulTy. . ' / ' * " ' > » 

• • - '. - " . » 
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Frustration. 

It was cua^ersoBie. Menu driven method get me bogged doim. /^ 

1) Found the menu selection to be helpful ^n the beginning but very tedious 
after this. * ' ' . ' . , . , 

2) ,U would be easier to use if all the terms, from particular vocabulary 

were* placed together. ^ 

The menus need to be improved. There should be an option to move directly to 
aqother set. There should also be more features for experien'ced searchers so 
that they could skip over the repetition of mem(s.' * , 

r think there is i lot of potential in this kind of switching system, 
hoi^ver, from r^y results I thijak it either needs some more work or it needs 
clear instructicms, with exanples, of its capabtll ties. 

I fhink it is a very good system and . something that is needed in on-line 
searching. . 

^t was laboricHJS and added very little value due to its limttations. 

favorable, except wheil system' and/or phone lin^ went haywire and I could not. 
get back on th6 right track by logging pf^ o" again. 

Ea'sy to use; good design; very helpful. 

The syst^ as currently implwiented is clisnsy and time-constlgilng. For 
acceptance in an on-line mode for help in structuring searches^ it would need 
to be streamlined ' and rely less heavily on its menu-driven features. The- 
capability to -update thesauri would have to be inplemented. Most of the adj.- 
leid produced only nonsense*. The^ system has potential of b^ing a useful tool. 

VSS gave me little, if any, ^ditional information.. K 
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16. «HAT DO YOU THIMC ABOUT SmX^CX SMITCHIII6 li^lCRAL? 

I think that 1€'s a very useful mechanism. Very often searchers forget to 
adapt their searches to other data base Indexing principles,, etc. Subject 
switching allows searchers to constantly keep aware of the fact that data 
ba^es are unique and that every tersi «u^t be checked. f ' 

I would prefer subject switching file 4U (DIALOG), and would gYadly pay 
for It. It Is a BRJch wore systewatic approach to database searching than 
plunging In wHh lists of free text terms. (If there Is leftover time, let me 
get back to It!) 

In theory, great — but this Is not It! 
Useful * ' ^ 

I think 1t>1s another unnecessary jargon term. You've built a gizmo. that looks 
through several thesauri In a very heavy handed mannisr.. IMder the rubric on- 
line thesauri," T like the idea. ' ' ^ t. 

No comnent. r - . * v 

Heeded. But the whole operation should be transparent to users. They should be 
able to, select a term and search a data base, without having to re-key the 
"select^" term. Because I was not fsiniar with searching the data bases 
that are part «f the Vocabulary Switching System', 1 prepared search questions, 
constructed pr*11es, aiid then used the Vocabulary Switching System to auonent 
or refine the terns I- had selected.- While this was not. a true test of the 
.Vocabulary Switching System, it did allow me to see n^ere the System could 
improve on the kinds and number of terms I had selected. 

Overall the Vocabulary Switching System was simple to use. I was not sure, 
however, of the meaning and use of the Vocabulary Switching ^tem cowrcands. 
These were displayed during searches, but I could not figure out exactly what 
each one meant. Had I had more time, rather than J«st two hours, I .could have 
experimented more with the procedures to see what all the Vocabulary S*1tcWng 
System could do. I felt pressed for time and perhaps this was the reason some 
of my searches were not too successful.* * * ^/ 4* 

The greatest asset of the V<K:abulary Switching System is the amount of time it 
saves since the correct search terms are selected for the searcher, fhen 
working with several vocabularies, lV is difficult, if not inpossible, to 
' remember the correct terns for each system. For* data bases with both 
controlled and free vocabularies, the Vwrabulary Switching System can lnf)f"ove 
search effectiveness since alternative pathways are suggested by^ome ,of the 

retrieved terms. * , ^ «. . 

One featui% that'iwuld have been very useful, even during this t^st, is a 
command that displayed which VSS vocabulary set and switching feature had been 
selected. Several times during the test I lost track of switching feature I 
had selected.. . - ' . ^ 

How about statistics on term searched (nimiber of times used in eacfi particular 
Job). . : 
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8-10 ,^ . ' 

No coRwent, • , 

Good Idea, would be nice to use updated thesauri. 

m 

It has to be done thoroughly or el s^- the searcher Is worse off. th^n left to 
his own devices and printed thesauri. ' 

No connent. » 

A very helpful tool for cross-disciplinary searching (e.g., education and 
psycho logy) . 

Might be useful In the future., , - 

No comient. 

Not too terrific In vocabularies 1 and 2 but It «1ght prove more useful In the 
others. ' 

Ho comment . ' - . 

No ccMment. • 

' . . . /■ 

Good theoretical Idea but devel(H)m«it requires a thorough knowledge of 
thesauri being manipulated In order to truly exploit the subjects. 

s 

As long as on-line databases use a controlled vocabulary, subject switching Is 
both useful and almost necessary for quick* easy on-liM searches. 

The Idea is a good one since we cannot always think of all the relevant 
subject headings for a given t<H)ic. ^It Is probably more useful In social 
science where "Searching must rely more heavily on descriptors/controlled 
vocabulary. " . ' 

Theoretically an excellent' idea, but I am left with the general in^jression 
that impl«Bentat4on, especially for a non-expert user, is extrwiely difficult. 

I thinie Its a "great , Idea, but either I wasn't *u^1ng it cqrrectly, it has 
limited applications, or It needs more woi^. . - 

Go<Sd idea — needs work. * • • - * - 

The definitions in blue sheets::|if .workbook mre not partficularly easy to 
understand ex.- "rel phrase", "wd match", 'etc. » . . > ^ - 

Synonyms, broader /narrower grea|^ idea — even* antonjffis yu^d be good. 

I like it when it .can be used I9 {thl^. databases djrectly: withoat re-keying 
terms. * •••• r 

I think it could te useful if it can, Jbe^ne cheaply <ntl fi|uickly. 

Generally a good idea — still' feef that .j>btcln1»« thesauri and consulting 
them win be necessary. " , ' 



It 1$ necessary. 

No counts. ' • ^ 

Valid' concept — should continue to be researched. 
6ood Idea, saves tisie, aids In searching. 

Subject Switching Is verjy valuable not only In doing searches on another data 
base, but also for lexicographical purposes. . ; ' 

Subject switching Is a niseful tool i*hen properly utilized^ but cofl|>utert2ed 
subject switching Is of questionable value when used to search multiple data 
bases. ; ' . ^ > 

In general, I think subject switching way be of assistance to the novice or to 
the unimaginative operator of machine retrieval equipment. I think the 
* knowledgeable 'operator would find VSS of little asslstjfftce. 

Subject sw4tcb4ii5, 4n^ner.a1, Is V useful tool for online searching, «n4* 
could be. very cost effective. T»« vocabularies m%t be k^t updated to provide 
%- # current material for high technology. 



Jftien an "Intelligent" computer Is available to handl^subject switching, life 
will be easy for all of us.. J feel that waiting for vocabularies to. be 
updated, printed, etc., makes the information outdated, so It will be 
greater benefit for the computer to be "the Impetus f or .the update- tffan the 
databases. ^ • ' 

We need It de^parltely but we need It better done. 

No coment. * ' <> 

Subject Switching Is of little use for the'tyN of on-line searching we do. It 
probably would be extremely usef ul , to on-line users , in highly technical 
operations. • ' 

Soocf idea if easy to use. • • . ' 
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S-12 . - ^ 

• ■ • . ♦ • 

I think It's difficult to do wel>. I think It requires more than simply 
"matching" te»*ms. In other words, I th^nk It takes some hunan thought to match 
up concept/ expressed differently In various thesauri* I also think 'that a 
majqr use for a VSS would for new concepts not -necessarily in a controlled 
vocabulary. Free-text suggestions are useful (e.g.. In a file such as BRS' 
TERM database. I found the TERM database concept more use^l and more 
f lex ible, ' mainly because of free text suggestions and because *feerm matches 
were determined by database Creator. 

' Pty questions fn using the VSS were'ialways: ^ 

• ^ - If I Already know the controlled vocabulary term, why search It 
. * . here? ^ 

- What am I learning that I cpuldn't set as easlljL In a print 
-thesauruisJ " I . - 

, . ■ ' . * 

-. How does th^s help me for new terminology or in searching fields 
I'm n^ fffliiliar 'with? 

- Do I really understand th MJurpose of this system? 
No ccwraent.' • • 

Battelle asked me 3 > years ago — I feel this is not one of - the conyuter's 
strong po^ts. » ■ 

Subject switching makes searching- across multiple databases much easier. The 
selection of the correct terms has' a signific||t Impact on the search results. 
Even free text searches would benefit vfromnhiT type of subject switching. 

Too cumbersomet. * v ^ * ^ . - ^ 

The. utility of ^ch a system; cannot be denied especfally if it could be 
incorporated Into database searching as a kind of automatic feature if and when 
the time coipe^ whep' slmuTtaneou^ dross-database searching is commonplace.* (1) 
61v0n the current capabilities of VSS, •! don'-t believe I- »w)uld have much 
occasion to use the system. (2) The menu-based command system is tood awkward 
add restrictive. (3) The current VSS^Insfructioa-^ookllt Is Incomplete and 
unclear, falling to expUIn the full potential; application, and capabilities 
of the system* , -Ss^j 

Wonderful Idea not done well w/VSS. "7^ 

I think it- Is a 'useful aid to anyone who needs to work with nua^ous sets of 
controlled vocabularies. " • ' 

interesting concept, but of doubtful utility. 

Good Idea. / . • 

It could be a valuable source of Information.* 

" "~ . ' . * - - 

It has potential for searchers who searih multiple databases but do not nave 
acpess to printed vocabulary. However, it would have to be quicker to use. 



G^d idea. . 

4 

Great idea* Needs more ability to work ifith phrases and scope ndtes>-character 
matching is still too primitive. . j >^ 

1) It is a useful aid to have when searching more than 2 databases- or when 
printed thesaurus is. unavailable*. ^ ■ * . 

2) Might be mol-e useful if it^was somehow Incorporated into existing features 
like Dial index In Dialog (or Cross in BRS) where postings of xhe terms 
aval lab te ar^ aUo given. ^ ' ^ 

3) Doubt I will utilize it af«!l if access toi it was based on the Sane pricets 
charged for the databases whose vocab are included. 

4) In suiaiiscy, I wowld. at best, use it occasionally as a pre-search tool if I 
could access it at a minimal charge. This is because I tend to mostly use a 
combination of contrplled and uncontrolled vocab. . . 

« 

Unless it is broadened to Include free-text vocabulary, it is not particularly 
valuable. * , 

I like Vnti concept. I think this system needs more work. ^ ^ ' 

In ^iew of the differences In vocabularies from one database to another, I 
f^el it wpuld be helpful to provide some aid to the searcher In automatically 
switching between the various vocabularies., ^ , • 

fto comment. ' * , 

In general, I think the idea has great potential, especlallji^ for searchers who 
routinely search numerous databases. The value of time saved alone would make 
subject switching worthwhile, I think, system needs refining,* and I would haye 
liked to have tested It in use iHth the databases (f.e., searchina a database, 
then logging on subjecf switching, then going on to next database.) 

-• • . . ' 

Mike It. . 

With the proliferation of on-Hne data bases, subject switching will become a 
4ie(:ess1ty. This is especially true for multi-discipline libraries. 

Good Idea, but needs much more work. 

i 



17. UUCNTS 

No connent. ^ 
No connent. . 
No cosment* 

♦ 

No conRent; i.. • 

\ ■ ' , . . ^ 

1 suspect that If . I had to |Jny for .VSS, i wouWn't use It. >is currently set 
•up. It produces entirely too RBich randan nolsg. It nay Be greeted more, 
lovingly by a librarian who has Httle-to-no Idea nhat a topic Is really about 
— but then soite of the false results meiy^mke her (or Hlol^ look like a real 
twit to thtf client. ■ - * ' 

r 

Ho comnent. 

- : % • ■ - ■■ ■ -, - - ■ ^ ■■ - ■■ . ■ - 

We would be Interested In receiving any reports generated a? a result, of this 
evaluation. Thanks for inviting us to participate. 

No coRRien^. 

fk) connent* ^ • 

Since -yom can't use BT/NT for PCSJi, you have to. use the printed thesaurus so 
ifse of VSS Is lifflited* • ' . • • 

Lack of entries for subheadings is a severe drawback. ^ • 

Inclusion of non >€SH terws is not useful If^ NT feature Is not avallablp — 

except to send user back to printed Thesaurus. 

VSS would.be easier to use if It gave one tt^ opportunity to Increase -the 
nuR^r of teriBS without having to review the n^le aenu. ^ 
Why are tenas duplicated with and without dates? 

Absence of \soiBe fairly (^vimis terms Indicates that the vocabularof was not 
buITt from the 1980 ISSH as was announced. 

If you €M't get NT/BT capability for the Life Sciences, at least. terns oo the 
sanie level sf^ould be retrieved.- ^ . " . ^ ' ;jt ' 




No consent* ■ \ 

\ ■ X 



No coRiiient.% 

(to coMent. , 

. , *" ' . - 

No coMsent. . . 

CROS and usually produce relevant files. Host of the fllesywKlch we use 
are easily man ipulaled using free text and descrlQ^ions^. As VSS/fs now I .woUld 
not use It or recomnend it. 

, \ - 

^No coflra^fft* 



Interesting experience as a test but one soon tires of menu-driven systems; It 
seens It would useful for foreign language vocabularies. 

I'pepsonal\y often find menus extremely tiring', particularly If one uses the 
progran often. For me. a two week trial would hive b^en more product4ve since 
I do r'elatiVely little searching. I need. more time to "play" with the system. 
I was unable to do any sfiatrching i^y last 2 days since I was Involved with an 
unexpected rush job (not searching). ^ , 

- / • ' * 

I was glad -to have an opportunity to try'VSS. It was^ an Interesting notion, 
but I found- it sowewhat cliwsy to use — and the menus were ????????? ! I use 
DIALOG' a lot and by comparlsoa, I 'found VSS to be very time-consimiing for the 
'amount, of 'response I obtained. In addition the capablTlty on VSS of recording 
thfe preceding- and followup terms, Ke-, (adj-lead and adj-word) Is interesting 
but generally Irrelevant since it applies only to descriptors (as opposed to 
an on-line index <Usplay where all the terms frcw the file that precede arW^ 
follow will be shownT- 

Mainly though, I was surprised by th«^ poor result si, especially in the science 
files. And, because this was a stand-alone capability, I had no way of knowing 
Hf results would have Improved and the whole file been searchjed, particularly 
In the-case of HASA. ^ ^ 
I m convinced that there is a need for thesaurus search CMabilitles on-line 
however, they cannot stand- alone as on VSS. Perhaps VSS in mod ff led form 
along with other search systeip capabiHties can be made into *a useful tool. 
But, T hav#^:oiiie away with the feeling, once again, that thesauruses in what 
ever form, ire not the best search tools because thejf do not stand alon^ well 
in today's searcHTngl ^ 

Some of 4ny selections^, e.g., electronic imH, may Mive been too cutfent for 
the experimental vocabularies, which I assume are stwewhat dated. 

My rather quick study of the instructions and inflexible u^e of the browse 
option may have led to the limited success I had with VSS, but any sort of 
practical application of this sort of 'thing would have to cope with naive ojse. 

You're truncating too far — I got Hopi for- homosexuality! 

No comment. ^ \ 

I would like to 'see a system like this cover even more files - also, I was 
very busy that week and not able to use it as Bwch as I wanted. I'm sure- 1 
would use this system regularly if it were not too expensive. 

Logon 'takes too- long, fast response time, menu format takes a lot of paper! 
See lettV. , ♦ 

No comment. 
No coBwent. 

The docimjentation waS quite skimpy, both, on-line and offline. Ifcch jiad to be 
learned trial and error fashion. 
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Ko cornment. ' . 

Lead tenas which are not posting tenas should ^ be Identified to permit 
retrieval of the ' posting term (X -references). Some 'stem terms deemed 
incorrect. I was disappointed. 

Uo comaent. ... 

The VSS system has a lot of Information but would take up a lot of time and be 
too expensive In Its present mode. 

The user - deigned switching feature Is poorly explained and Illustrated. 

Machine retrieval of Information as a Judpaent matter.^ creative mind Is a 
great help." Search of title and abstract, for. unique words or phrases combined 
with truncated terms in the indexing fields Is a direct approach which will 
result In pertinent citations. These provide a. r^ady _ display of Indexing^ 
. fields as well as Identifiers and special niwerlcal designators. This 
approach does pre suppose all data bases will provide text search capability 
and that emphasis Is placed on good title, title extensions or supplements, 
and on a well written abstract. 

The subjects selected for the four test^ provided few specific subject terms. 
The Broader /Narroi^r Tgrms feature produced little specif 1c .output: The most 
helpful Subject switching feature was Browse . ^ 

£SA-QUEST's ZOOM feature IS compaJ^e to this "program In j)laces. 

\ ' ( r ' : . 

Whatever youMl charge will be B0i»e than Td want to paof to perhaps save a few 
moments of manual checking. / ' * .- • 

/ * >^ " ^ . 

* Nevertheless I must conclude that VSS In Its present form leaves a Ibt to be 
desli^ed If. we are to consider Its u^e as an attachment to DI/IU.OS or to an on- 
line catalog which has access to more than one library's catalog. My reasons 
for this >«re as follows f 

1. VSS In Its^^resent form Is mainly menu . driven, not allowing much 
flexibility other than those pre-ordained;' the change from pne set of 
thesauri to another \% laborious; the options do not match the, 
characteristics of the thesaurtjior those of the searching tenns. 

, 2. The browse option, the easjest to access, has an ordered output which 
prevents a thesaurus display (rejated terms, BT-MT) until all the stowing 
lists are given; oftentimes these are way off target and the searcher imist 
cycle back, gfccrease the listing length, go through all the st«ns again, 
etc., befon^lettlng tp the thesaurally linked terms. When the other 

t options are used and there, is not an identical-" match, "search Is not 

SUCCGSS^ul • * * 

3. I • was very surprlsoL to see "what ;df ^he various thesauri you had NOT 
loaded, e.g>, the MEsi tree structure. This knocked out In one. f el r swoop 
'all the BT-irr relationships. Surely you could have done something lltee a 
table look-up to preserve those relatlonsMps. The sane thing Is true of 
other thesauri tJfet use ciitegory niB^ers %q exhibit hierarchies In outline 
fashion rather than only one level at a time. The next version of VSS would 
have- to remedy this weakness. I think the. BSI ^?0OT program does better. / 
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4,, You never show scope notes, so valuable as "definitions". 
No coiaBient'. * * ^ 

No consent* 

No coiwent. i* * 

No comment. 

.' ' _ . , 

Mo conment* 

Nocc-ent.. ■ *" 

The present tutorial approach used In V^S should be supplemented with a more 
advanced verslijn In which several tenas and combinations of options could be 
searched. This would provide faster switching and make VSS use wore cost 
effective. This would also enhance its use for thesaurus construction. 

> r - ; ; ' • . • 

Nopo^pent. 

/«±hough the "tefhs tjrpe" relationship Is given for retrieved teros, there is 
no\cope note, and when searching for vocabulary on a t<H>ic about which I had 

. very little personal knowledge, it was difficult to evaluate **ether a 
particular retrieved term was closely related to original tefw or whether 
it was only a remote, and therefore useless, relationship that would only 
confound the results of a bibliographic on-line search. Therefojee, I wuld 
have had to spend a certain «unt^of time researching these ui|f^lmr 
retrieved teHnk in order to determine their potential contribution to an on- 
line search strategy. Since I would have speot time researching database 
thesaurus and other references as a mattfci of routine,*, before coi^iling a 
search strategy, this additional searching on VSS would only have represented 
redundant effort. Another specific criticisiB I have of VSS is the fact that 
one is severely restricted in retrieving pertinent terminology since one must 
initially enter a term or word that- is r^resented in at least one of the 
vocabulari€S. Unless' you hap^n to select a word that is part of an actual 
controlled vocabulary you are wit of luck. T?« system does not appear to have 
any provision for diriecttng the user to terms that would be more P»'OTitable. 

' From my personal experience In thesaurus construction we att«!?)t to keep USED 
rm references at a minimum both to save space in our thesaurus and so as not 
to clutteV the thesaurus with what is potentially an infinite niaber of 
synonymous expressidhs for terms. I imagine that other thesaurus constructors 
have a similar policy. If so, this sets a built-in limitation for VSS. In 
short, VSS output is only as good and extensive as the vocabularies that 
cpmpose it, and it cannot serve as a substitute for research in other sources 
for terminology. V ' 

No comment. • ' 

Please seek .nbre funding and work more on making the system responsive to 
someone who does not know how to do. preterminal research (using thesauri). 

... 

No comment. ^ 

20-2 



S-18 

fto conment. ^ ' 

No cofMsent. 

This fqrm (#11) -led tne t'o believe that after locating terras I would then do 
the search -and-find out whether or not .tfi^ search was successful. It Is 
dlfflculfto evaluate the usefulness of the terins without doing the search. 

I -diSn't realize you«would go down at 5 pm, which Is about the tlme^I signed 
on* 1 ♦ . . ' 



No coments. ' . 

' ... . * 



No coflsieht. 
No coRBient. ' . ' 

For something like C.A. to be useful, you WDuld need tp have the synonM»s of 
the various cherplcals and trade nms avallable^^ 

1) In the login Instructions, It would be helpful to underline what /the user 
must enter (the actual - words an<^ syslitols) rather than the phrase "you 
enter". . . •> 

2) Since the options available for each vocabulary differ. ^ wdSid be helpful 
If one was presented only with those options that applied to the vocabulary 
chosen, rather *than the entire menu of cations, ^/one of whjch do not apply. 

3) Since I had a tendancy to "want to search all options (browsing,, theij 
broader and narrower terms, ^ *c.) for a particular vocabulary tena^fore 
going on to another term. It wight be hflpful^to consider that sequence 1n 
searching rather than the one presently used^ requiring, that one r^ter. 

-the vocabulary term each tla^' a nmi t^tlon Is chosen. 

4) in iny evaluation of the* search results, I assiaaed certain tierms were 
applicable to tsy search thiHigh, In fact, I was not sure of soae of them, 
since no' definitions were available oni^llne. It might be helpful, tiMMigh, 
to provide the possibility' of seeing the scopte notes on-line, for those 
terms for which the meahlngs are unclear. . 

5) In the sample searches. In the booklet, Jt seems a step was left out* after 
one chooses the max1mi» nim^er; One then'ipust enteral or 2 or 3 or term or 

, conwana" and It was confusing 4t first whit the #s 1, 2. #nd J^eferred ta 
until I. had done an actual search ai^ found there was a step before this 
which g^^e a menu of options from which to' select. • . 

6) A better Introdu^lqj^ to VSS would helpful, to explain Its function. 

No comment. , ^ * 

If a searcher makes a typo or selects wrong menu # there should be an. easier 
way to go back and correct It. I got fflTustrated trying to get #5 recognized so 
I could start over. The problem mayinave been poor phone lines and not the 
system. / 



Suggestions for lofirovement : . * - - ' 

-^a. Verify the spelling of the user by returning the search statement 
b. Permit direct change of switching feature on any given term. 

m 

I ■ ' , ' ' ■ 
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* * ♦ 

^, Switch feature (6) was very helpful; the user can confine the 
features that are wost apprqjrttte for h4s/her needs 
and the vocabulary. ^ " I 

d. Would It be possible *o' move up and dowrt In the MESH' tree- 
, structure? That feature iiould be more tielpful than adjective 
. terms. 

«* Ho comment. 

Thanks fpr -Including BIOSIS In the study. Looks Itke W wITT have to 
%dex more thesaurus-like! I think the bes^ "se^ch^ng" happened on t 
PSYCHABS and MEW-INE vocabularies. * ; 
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16. Mmc of row iHroMftt siagU dau k«M SMrctes mj^t ^ 
you pwfon ftctoM Mltiflo data baoM If fon bad a a^Joct ^ 
atrieabl^ cmpAllltjl ....... \ ^ . . i . . ^^^^^ 

n. lAiat alngla factofr vovld'a^ inflttoira d ac iaio n to oae «^ 
Jact avitdiiias to a glvM aa«rcli. (Cbadk only <pa} 

Coat of aaii« a^j^t anltehi^g • • • 

of data baaaa aaadad for tte aaa rfffc ••••••• • *^ 



Tte diffaraaeaa la teta baaa iracdbala^iM ^ 

OtKar (Fiaaaa SpacifF) . • • I_ 

18. ApproKlaataly liov aocb would fqa im willias to for a n^Jact 
' avitchiiig eapabil^» and tte adda/coat of ttAra^.ootputt (Chack 
{75j oi4y om) ^ i • 

f ' ' i' - . 

. Mothi^ at all ^. _ 

Up to $ 5 poft adarcli • • • • • • • _ 

*i i - $XD par aaarah ^ • 

$11 - $20 par aaarch .. ...... . . . . . - . . .V. 

$21 - $3a.jper aaarcli /"^ 

Over ^31 par aaareb - ^ 



IS. lov many data ttaaM nould a aiagla raguira bafora you ipould ^ 

uaa tte aubjectg^tching capability? (Cback mly cme) . 

TVo /*** - -- 

threa ^. . . . . .' ^ 

3 

Four or won • • ■ * 

•# 

Conenta: 
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1. lodicata tte typa of orgaalaatioii fox aiiieb yoi^ worki • ^ 



Ifalyaralty or c^pllage • • . • ^ • . ^. . . , 

For-profit Industry 



•j:- 



Sonrprof it<- iaduatry t aaaoc i a t t oa , or raoaaro^ ioafituta . .• . 
eo».™t .8«iiy . . .'. ... , .,. 



Public library . i ........... \ * 



Mbar. (Fiaaaa Spacify) 



2. Iteicata wtetl^ /te wa » «d uaar i&forMtioo producta or aa 
iotarsadiary peffafaitoi aaar^^ for Md Moras 



2 

lata i i>iid i a i . t^*,. «. .. _^ ^ 



3| Of tte rad uaera for aten yM ftprfpra «Barc^» appr«^Mtet«ly 

parcaatagb aras * 

a. Jjmm^TUX wdtt . . . . / 

^ b. ^iaatiata * \ ' \ , . . . . 

c. Faculty ... .• ....... .*. . - ^ 

d. Studaata . . « % • • * fi fri 

a. Llbrarlaoa/bibli^raptera ......... * • .'^ > .^^ 



Fiaaaa aatlsatat Ki^ar of obwaiBt' ^ Oaliiia 



f. Otbar ^FleaM Spacify) . - ' 



Saarcbaa/ao. ' Hra/ao.. ^ po#ti/ao. 



4. Your indivi^ial aaarch ' L / 

activity . a^i b»^^ c. f 

5. Tcmr orga nl r a tioaa * a ■/ 

-i-rcb activity ...... .^.^^j,,^^: 1.^^ 

6. ApproKiaately «^t /parcantaga of tba tearctea foa parfoxa \ra paid 
for hft (Anawats abcmld- total lOOK) « 

• * "• ■ 

a. ExkS u^rs^ budget* deportaeat or project dollara. 

b. Library or iaforaatdoo crater budget 

c. Coabindtioo of tte abwa ' • 

d. Otbar (Flaate Spacify) ^ > 



mm* 
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.; • ' • . 

ApproxiMtcly lAftt p«rcnt«g« of thm Muctes fou pmctnm hmn thm 

. / * * ^ • 

hm Stajr vitlUa * giMb CMt f Icnr* . • > % 

71-7J 

c* Dm only a giv«o <oyj»fcaoo or ^tm bosoa • • • X 

«• Stay vithia a givaa coaaact ttm • • • • ^ • J % • • • • • Z 

I, ftaaaaatly 2. go^tfaiea a. Iafroottaatly 
a! iocl^aod fiMLOG ' " 

n — . n — • IT 

b. .roc OBBIT ^ . 

^ . nri — n — n : 

c. BiS • 

— ^ ^ — — 

— nrs 13 — - — Tt"* — 

o. DOC * , . 

IS — n — — rr — 

f, BEOOH HhSAIi - ■ „ ^ 

TT TJ -'If . ,\ 

^XLL (HLX) - 

Otbarai . 

• — — „ „ - — ^ 

• ^ ; ^ ^ ^ 

Approxiaately vbat ^wrcaataga of yoar aaoxbea wa 

(AnsMra abould total lOQH) ^ ^ - 

Slngla data baaa saarcbM * X 

b. Ifciltialo data baaa 'aaarcbaa involvii^ 2 data baaaa ^ . . ; , > 

c. Nultlpla data baaa aMrcbM iavolvlag 3 data baaa^ . . . • -j^^^ 

d. Ibatip^e data baaa aaarcbaa nnvolAng 4 data baaaa • . . . -7^=^* 
'et Don't kaov ^ % 

fiA^T ahal coodftioaa ara yoor aaarelMs limited to ima data baaa 
^ {iy^»ar yfMir cboicaa 1, 2» ate. la ordar of iaportmca). 

a. Quary doaan*t rai|uira aora axbauatlva a^afcH . 

b* Too difficult to uaa otbar, laaa fasiliar data baaaa , . • 

c. Too tiM conaoalng to do Miltipla data baaa Marcb « • • « 

d. Too coatly to do mltipla data, baaa aaarpb 

n 

a, ' Too aany data baam availabla to kaoif t^eb oaaa to aia , 

^ ^ — n 

f . Othar data bal^ mt availabla to ay orgaaiaation • • • • ♦ 

g, Otbar (Flaaae Spacify) _ • • 



H-3 



■ u. 



If you ara aaarehiag oa*t«o or vori data bMOa* la abiab. gaaaral aub- 
je^t araaa la mltipla data'b^ ^aaarclSli^ ttoat iia^ii|.t (lhaibar 
:^ior f Irat cbraa eboicM 1, 2, aad 3) , ^ - . 

EdwatlM 



Oioaiatry 

Baaiaaariaa 
*Xlfa SciaaCM/Madlciaa 



AsricultuTtf ■ 
Baorgy 
Bmri r o iM W B t 
Psychology 

Otter (riMM SpMlfy) 



12. 



If you igm MMebiag two or aero data baaaa. uhlch data fcaaao d<» 
you '«oat f ra^MBtly caAtStH 



tmo ^ta baaa 


1 (12-^17) 
• 


mmm of Data Baa 


1 (a.g; .GOI^^IBSX) Rot Satriaval Syatea 

" 



Tbraa ,data >aaa aaarcb (18^2$) 



c. Four or tora data baaa Marcb (27«-80) 



/ ^ 

13. How of tm do you aaarcb with tba fol loving: 



i 



a. Coatroll^ vocabulary taraa ooly % 

b. CMtrollad plua frw taat taraa X 

c. Fraa taat tarm oaly . . , « X 

14. Indicata your prafarasca balov (Plaaaa rank '*l**.for *^mt prafarred*** 
**2^ for aaxt soat praferrad*^ aod **3** for **laaat praf arrad*"} « 

a, CoQtrollad voc^lary tarM only • . ' 

• S3 

b, Gootrollad plua fraa tast tarM 

- , 

c» Fraa text terw mly 

15. Givaa a ai^Jact switcbli^ capability aach aa that daacribad in 30 
tba covaring lattar, la what parcaataga of your praaaat aMrcbea 

do you tbiak you night uaa it? ^ • « % 



